Event‐related potentials evidence for long‐term audiovisual representations of phonemes in adults
- 25 November 2021
- journal article
- research article
- Published by Wiley in European Journal of Neuroscience
- Vol. 54 (11) , 7860-7875
Abstract
The presence of long‐term auditory representations for phonemes has been well‐established. However, since speech perception is typically audiovisual, we hypothesized that long‐term phoneme representations may also contain information on speakers' mouth shape during articulation. We used an audiovisual oddball paradigm in which, on each trial, participants saw a face and heard one of two vowels. One vowel occurred frequently (standard), while another occurred rarely (deviant). In one condition (neutral), the face had a closed, non‐articulating mouth. In the other condition (audiovisual violation), the mouth shape matched the frequent vowel. Although in both conditions stimuli were audiovisual, we hypothesized that identical auditory changes would be perceived differently by participants. Namely, in the neutral condition, deviants violated only the audiovisual pattern specific to each block. By contrast, in the audiovisual violation condition, deviants additionally violated long‐term representations for how a speaker's mouth looks during articulation. We compared the amplitude of mismatch negativity (MMN) and P3 components elicited by deviants in the two conditions. The MMN extended posteriorly over temporal and occipital sites even though deviants contained no visual changes, suggesting that deviants were perceived as interruptions in audiovisual, rather than auditory only, sequences. As predicted, deviants elicited larger MMN and P3 in the audiovisual violation compared to the neutral condition. The results suggest that long‐term representations of phonemes are indeed audiovisual.Keywords
Funding Information
- National Institute on Deafness and Other Communication Disorders (R01DC017017, R03DC013151)
This publication has 55 references indexed in Scilit:
- Seeing the initial articulatory gestures of a word triggers lexical accessLanguage and Cognitive Processes, 2013
- Human Visual System Automatically Encodes Sequential Regularities of Discrete EventsJournal of Cognitive Neuroscience, 2010
- Visual Object Representations Can Be Formed outside the Focus of Voluntary Attention: Evidence from Event-related Brain PotentialsJournal of Cognitive Neuroscience, 2010
- Deviance detection in congruent audiovisual speech: Evidence for implicit integrated audiovisual memory representationsBiological Psychology, 2009
- Visual Mismatch NegativityJournal of Psychophysiology, 2007
- Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusionNeuropsychologia, 2007
- Audiovisual Events in Sensory MemoryJournal of Psychophysiology, 2007
- Probability‐independent and ‐dependent ERPs reflecting visual change detectionPsychophysiology, 2006
- Generalization of the generation of an MMN by illusory McGurk percepts: voiceless consonantsClinical Neurophysiology, 2004
- Hearing lips and seeing voicesNature, 1976