Multimodality is an important term in understanding
perception and cognition.
The term modality can be understood as a channel
of sensory information.
It's a broader term than the five senses,
and includes, for instance, vision, hearing, balance,
and proprioception.
We use multiple modalities to get information
about the world.
Different modalities may provide the same information.
For instance, either by hearing or by seeing a balancing ball,
you'll get an indication of the size
of the ball, how hard it bounces,
and the distance to the ball.
In most cases, we receive information
through multiple modalities at once.
And the modalities confirm each other,
contributing to our perception of the environment.
But contradictory information to different modalities
may lead us to perceive information that is not
present in what we perceive.
A good example of this is the McGurk effect.
Watch me while I say, gah, gah gah, gah, gah, gah, gah.
What did you just hear?
Most likely, you perceived the sound, dah.
But in fact, you heard the sound bah
on top of a video of me saying gah.
Let's watch the video again without sound.
And now only listen to the sound without watching the video.
gah, gah, gah, gah, gah, gah, Finally,
let's play the combination again.
gah, gah, gah, gah, gah, gah.
Most likely you perceived the sound dah.
But in fact, you heard the sound bah
on top of a video of me saying gah.
The multiple nature of our perceptual system
may explain why we easily project features describing one
modality onto another modality.
For instance, projecting sound features into a shape,
Wolfgang Kohler demonstrated this as early as 1929
with the two figures takete and maluma.
When asked to match the words to the figures,
most people said that the curve the figure on the left
was maluma, and the jagged figure on the right takete.
Somehow our understanding of the word takete
with its short consonant sounds match well
with an abrupt jaggedness of the right figure.
And the more prolonged vowel sounds on maluma match
with the bloated figure on the left.
To sum up what we have learned in this video,
sound perception is more than sensing vibrations
in air pressure.
When perceiving the sound, we associate
the sound with some sound producing action.
For instance, impulsive sounds such as these, [SOUND EFFECTS]
are associated with some impulsive
action, a fast transfer of energy from one object
to another.
We also make up our mind about the sounding object,
it's size, shape, material, location, and so forth.
And although the sounds produced by a single object
may be very different, we're able to attribute them
all to the same objects, like these symbol sounds.
[SYMBOL SOUNDS]
We have seen that sounds may give
associations of properties of other domains such as shapes.
And we know that musical sound may cause
strong emotional reactions.
This demonstrates some of the highly complex mechanisms
involved in perception.
And in the following videos, we will
have a closer look at the implications
of this in music research.