"Mono-sensing" and Understanding Speech
Published February 12, 2009 @ 05:00PM PT
Read my lips: Using multiple senses in speech perception reports on an article "Speech Perception as a Multimodal Phenomenon" (Current Directions in Psychological Science Volume 17, Issue 6, Date: December 2008, Pages: 405-409 Lawrence D. Rosenblum). The article discusses how comprehension of spoken word is likely something that requires input from more senses than just hearing; vision, and even touch may play a large role in how most people make sense of spoken word.
Take a gander at the instructive video below (not captioned, but the phenomena the video captures is described in the link to the report above).
What did you think the person was saying? Did you get it correctly, or were you fooled by the "auditory illusion?"
While it's not part of the criteria for any ASD, many people on the spectrum (myself very much in this category) report difficulty processing more than one sensory channel at a time. Understanding sound and image at the same time is in fact so hard for me that I use closed captioning when I watch movies and television shows--I can process just the visual and let go of any attempt to make sense of the audio. And believe me, I have much more acute hearing than I'd really like... When I have to understand what people are saying, even in a quiet setting where I can separate their voices easily from background sounds, I have to "turn off" my vision and all other senses in order to process the word-sounds at all.
When I watched the video, I was not fooled by the illusion. Perhaps because, even with eyes open, I was not seeing (in the sense of processing the visual stimulus) the person's mouth move. Did this happen to anyone else who has "mono-processing" with their senses?
If understanding speech is indeed how Rosenblum describes--a multimodal experience--what might that mean in terms of an explanation for why some people on the spectrum have difficulty understanding or keeping up with spoken language?
Share this Post
Related Posts
-
Cyborg Meltdown Alert Device?
-
Keepon the Robot Interacts without the Overload
-
The Dynamics of "Communication"
Comments (10)
Comments on Change.org are meant for further exploration and evaluation of the ideas covered in the posts. To that end, we welcome constructive comments. However, we reserve the right to delete comments that are offensive, abusive, or off-topic; that contain ad hominem attacks; or that are designed to subvert or hijack comment threads rather than contribute to them. Repeat offenders may be permanently removed from the site at our discretion.
Facebook
Twitter
Digg
StumbleUpon
Delicious
Email


















I am profoundly deaf and use a cochlear implant, as well as a hearing aid. Listening to this video, I thought I heard "the, the, the..."
When I was speech reading (also known as lip reading), I could not catch much because the speaker's facial affect was blank and abnormal--as far as speech reading goes anyway. That noted, I guessed the speaker had to have been using a sound like /g/ since his lips, tongue, or teeth did not touch for a /b/, /v/, or /th/ sound.
I was surprised to learn that the actual sound coming from the speaker was "ba, ba, ba..." as I had pretty much bet that I would not be fooled.
Interesting video with a good point. Thanks for sharing Dora.
Posted by Elesia Ashkenazy on 02/12/2009 @ 06:19PM PT
You must be signed in to report content.
I do kind of lip read a little when I watch TV and movies, and when I turned off my hearing and just watched the video, I couldn't believe the person was actually mouthing "ga." The "a" sound seemed there but I didn't detect the proper throat/jaw stuff for the "g" sound. I also found the speakers eyes especially disturbing--I can usually manage eyeballs OK when they're on the screen, that that guy was pretty unbearable. I'm glad I'm not the only one who found the speaker strange!
Posted by Dora Raymaker on 02/12/2009 @ 06:37PM PT
You must be signed in to report content.
Interestingly, in all the McGurk Effect videos I've seen, I hear something that is neither "ba" or "ga"; in fact, it's not even a sound from English. It's similar to "ga", but pronounced much further back in the throat.
Apparently I really do fall back on lip-reading, since the sound is still much closer to "ga" than "ba"...
Posted by Cody Boisclair on 02/12/2009 @ 07:00PM PT
You must be signed in to report content.
I heard the ba sound with my eyes open and closed. I got nothing out of the video with no sound.
Posted by Jen Rosenblum on 02/12/2009 @ 08:47PM PT
You must be signed in to report content.
I just watched a few more videos of this on youtube. In the video with the hairy guy, I heard ga. In the Hear With Your Eyes video, I heard the da the time when I saw the person, and the be when I didn't see him. Is the person supposed to have anything to do with it?
Posted by Jen Rosenblum on 02/12/2009 @ 08:53PM PT
You must be signed in to report content.
Wow, interesting. I wasn't fooled by the illusion either! I had a hard time even seeing the illusion when it was explained, even. Mostly when I look at faces I think I tend to focus on the shapes of features, on colors of hair, of hair patterns, etc. And I also watch thing with subtitles on -- when I discovered that I was shocked at how much more "plot" I picked up on in movies, etc.!
Posted by Anne Corwin on 02/12/2009 @ 09:15PM PT
You must be signed in to report content.
All I heard whether I was looking or not, was 'aah' 'aah' etc. As far as I can tell, if it is 'ba' or 'ga' (as in, hearing any consonant with the 'aah'), that is only because he closes and opens his mouth while saying the 'aah' sound. And I can't even imagine how anyone comes to 'da'.
Posted by Norah vd Stel on 02/13/2009 @ 02:15AM PT
You must be signed in to report content.
I was hearing "ah bah ah bah bah bah bah". With the guy lips I was stuck on think he was saying "dah".
And I have a mild auditory processing disorder... mostly with filtering out background noise. Maybe I over compensate?
Posted by Erin Monk on 02/13/2009 @ 03:31PM PT
You must be signed in to report content.
I hear aah with my eyes open and ba with my eyes closed.
Posted by Holly Banfield on 02/15/2009 @ 07:15PM PT
You must be signed in to report content.
Although I understood the explanation intellectually, I was unable to observe any of the effect they were describing.
This is probably wy I have such great difficulty with unscripted and casual conversation.
Posted by William Brown on 07/04/2009 @ 10:12PM PT
You must be signed in to report content.