Fusion and combination in audio-visual integration
Authors:
- 27 November 2007
- journal article
- Published by The Royal Society in Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Vol. 464 (2090) , 319-340
Abstract
Language is essentially multi-modal in its sensory origin, the daily conversation depending heavily on the audio-visual (AV) information. Although the perception of spoken language is primarily dominated by audition, the perception of facial expression, particularly that of the mouth, helps us comprehend speech. The McGurk effect is a striking phenomenon where the perceived phoneme is affected by the simultaneous observation of lip movement, and probably reflects the underlying AV integration process. The elucidation of the principles involved in this unique perceptual anomaly poses an interesting problem. Here we study the nature of the McGurk effect by means of neural networks (self-organizing maps, SOM) designed to extract patterns inherent in audio and visual stimuli. It is shown that a McGurk effect-like classification of incoming information occurs without any additional constraint or procedure added to the network, suggesting that the anomaly is a consequence of the AV integration process. Within this framework, an explanation is given for the asymmetric effect of AV pairs in causing the McGurk effect (fusion or combination) based on the ‘distance’ relationship between audio or visual information within the SOM. Our result reveals some generic features of the cognitive process of phoneme perception, and AV sensory integration in general.Keywords
This publication has 46 references indexed in Scilit:
- Temporal window of integration in auditory-visual speech perceptionNeuropsychologia, 2007
- Is neocortex essentially multisensory?Trends in Cognitive Sciences, 2006
- Seeing and hearing others and oneself talkCognitive Brain Research, 2005
- Primary auditory cortex activation by visual speech: an fMRI study at 3???TNeuroReport, 2005
- Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjectsPerception & Psychophysics, 1997
- Temporal constraints on the McGurk effectPerception & Psychophysics, 1996
- Integral processing of visual place and auditory voicing information during phonetic perception.Journal of Experimental Psychology: Human Perception and Performance, 1991
- The motor theory of speech perception revisedCognition, 1985
- Visual influences on speech perception processesPerception & Psychophysics, 1978
- Hearing lips and seeing voicesNature, 1976