Neural Mechanisms Underlying Cross-Modal Phonetic Encoding
Open Access
- 20 December 2017
- journal article
- research article
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 38 (7) , 1835-1849
Abstract
Audiovisual (AV) integration is essential for speech comprehension, especially in adverse listening situations. Divergent, but not mutually exclusive, theories have been proposed to explain the neural mechanisms underlying AV integration. One theory advocates that this process occurs via interactions between the auditory and visual cortices, as opposed to fusion of AV percepts in a multisensory integrator. Building upon this idea, we proposed that AV integration in spoken language reflects visually induced weighting of phonetic representations at the auditory cortex. EEG was recorded while male and female human subjects watched and listened to videos of a speaker uttering consonant vowel (CV) syllables /ba/ and /fa/, presented in Auditory-only, AV congruent or incongruent contexts. Subjects reported whether they heard /ba/ or /fa/. We hypothesized that vision alters phonetic encoding by dynamically weighting which phonetic representation in the auditory cortex is strengthened or weakened. That is, when subjects are presented with visual /fa/ and acoustic /ba/ and hear /fa/ (illusion-fa), the visual input strengthens the weighting of the phone /f/ representation. When subjects are presented with visual /ba/ and acoustic /fa/ and hear /ba/ (illusion-ba), the visual input weakens the weighting of the phone /f/ representation. Indeed, we found an enlarged N1 auditory evoked potential when subjects perceivedillusion-ba, and a reduced N1 when they perceivedillusion-fa, mirroring the N1 behavior for /ba/ and /fa/ in Auditory-only settings. These effects were especially pronounced in individuals with more robust illusory perception. These findings provide evidence that visual speech modifies phonetic encoding at the auditory cortex.SIGNIFICANCE STATEMENTThe current study presents evidence that audiovisual integration in spoken language occurs when one modality (vision) acts on representations of a second modality (audition). Using the McGurk illusion, we show that visual context primes phonetic representations at the auditory cortex, altering the auditory percept, evidenced by changes in the N1 auditory evoked potential. This finding reinforces the theory that audiovisual integration occurs via visual networks influencing phonetic representations in the auditory cortex. We believe that this will lead to the generation of new hypotheses regarding cross-modal mapping, particularly whether it occurs via direct or indirect routes (e.g., via a multisensory mediator).Keywords
This publication has 67 references indexed in Scilit:
- An fMRI Study of Audiovisual Speech Perception Reveals Multisensory Interactions in Auditory CortexPLOS ONE, 2013
- Positive Outcomes Influence the Rate and Time to Publication, but Not the Impact Factor of Publications of Clinical Trial ResultsPLOS ONE, 2013
- Neural restoration of degraded audiovisual speechNeuroImage, 2012
- A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusionNeuroImage, 2012
- FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological DataComputational Intelligence and Neuroscience, 2010
- Continuous Perception and Graded CategorizationPsychological Science, 2010
- Primary and multisensory cortical activity is correlated with audiovisual perceptsHuman Brain Mapping, 2010
- The Role of the Posterior Superior Temporal Sulcus in Audiovisual ProcessingCerebral Cortex, 2008
- Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusionNeuropsychologia, 2007
- Hearing lips and seeing voicesNature, 1976