Audio-visual onset differences are used to determine syllable identity for ambiguous audio-visual stimulus pairs
Open Access
- 1 January 2013
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Psychology
- Vol. 4, 48344
Abstract
Content and temporal cues have been shown to interact during audio-visual (AV) speech identification. Typically, the most reliable unimodal cue is used more strongly to identify specific speech features; however, visual cues are only used if the AV stimuli are presented within a certain temporal window of integration (TWI). This suggests that temporal cues denote whether unimodal stimuli belong together, that is, whether they should be integrated. It is not known whether temporal cues also provide information about the identity of a syllable. Since spoken syllables have naturally varying AV onset asynchronies, we hypothesize that for suboptimal AV cues presented within the TWI, information about the natural AV onset differences can aid in speech identification. To test this, we presented low-intensity auditory syllables concurrently with visual speech signals, and varied the stimulus onset asynchronies (SOA) of the AV pair, while participants were instructed to identify the auditory syllables. We revealed that specific speech features (e.g., voicing) were identified by relying primarily on one modality (e.g., auditory). Additionally, we showed a wide window in which visual information influenced auditory perception, that seemed even wider for congruent stimulus pairs. Finally, we found a specific response pattern across the SOA range for syllables that were not reliably identified by the unimodal cues, which we explained as the result of the use of natural onset differences between AV speech signals. This indicates that temporal cues not only provide information about the temporal integration of AV stimuli, but additionally convey information about the identity of AV pairs. These results provide a detailed behavioral basis for further neuro-imaging and stimulation studies to unravel the neurofunctional mechanisms of the audio-visual-temporal interplay within speech perception.Keywords
This publication has 51 references indexed in Scilit:
- The sound of sizeNeuroImage, 2012
- Transitions in neural oscillations reflect prediction errors generated in audiovisual speechNature Neuroscience, 2011
- Task‐irrelevant visual letters interact with the processing of speech sounds in heteromodal and unimodal cortexEuropean Journal of Neuroscience, 2008
- Enhanced Visual Speech Perception in Individuals With Early-Onset Hearing ImpairmentJournal of Speech, Language, and Hearing Research, 2007
- Factors influencing audiovisual fission and fusion illusionsCognitive Brain Research, 2004
- Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture InformationJournal of Cognitive Neuroscience, 2004
- Integration of Auditory and Visual Information about Objects in Superior Temporal SulcusNeuron, 2004
- Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortexCurrent Biology, 2000
- Speechreading and the structure of the lexicon: Computationally modeling the effects of reduced phonetic distinctiveness on lexical uniquenessThe Journal of the Acoustical Society of America, 1997
- Activation of Auditory Cortex During Silent LipreadingScience, 1997