Visual Gestures of the Head and Eyebrows Support Prosody Perception for Individuals with Cochlear Implants
Open Access
- 1 January 2026
- journal article
- Published by SAGE Publications in Trends in Hearing
- Vol. 30
Abstract
Visual gestures, especially head movements and eyebrow raises, are time-locked to acoustic cues during the expression of spoken prosody. The present study examined the role these visual cues play in prosody perception, particularly for individuals with cochlear implants (CIs), who often experience challenges understanding prosody due to reduced access to pitch cues. A vocal mimicry paradigm was used to obtain granular, objective measures of prosody perception through acoustic analysis of mimicked sentences. Stimuli consisted of audio-visual recordings from one talker that captured naturally occurring variability in the expression of auditory and visual cues to word focus. Participants mimicked these natural recordings, as well as prosody-transplanted stimuli that allowed us to isolate the influence of auditory or visual prosody cues while holding the other modality at a neutral level (broad focus). Participants converted visual prosody cues into acoustic correlates of prosody in their mimicry, repeating acoustically unfocused words with higher F0 and intensity when those words were paired with video containing head and eyebrow gestures. This visual influence was stronger for participants with CIs than an age-matched group of typical-hearing listeners. CI participants who were less successful at acoustic mimicry tended to be more influenced by visual cues. These results indicate that CI listeners compensate for degraded auditory cues by integrating visual gestures into their perception of spoken prosody, potentially highlighting new targets for multisensory counseling or training.Keywords
Funding Information
- National Institute on Deafness and Other Communication Disorders (F32DC021847)
- National Institute on Deafness and Other Communication Disorders (R01DC017114)
- National Institute on Deafness and Other Communication Disorders (R21DC021495)
This publication has 78 references indexed in Scilit:
- Auditory-Visual Interactions Subserving Goal-Directed Saccades in a Complex SceneJournal of Neurophysiology, 2002
- Coding of the fundamental frequency in continuous interleaved sampling processors for cochlear implantsThe Journal of the Acoustical Society of America, 2001
- Articulatory Correlates of Prosodic Control: Emotion and EmphasisLanguage and Speech, 1998
- Prosody in the Comprehension of Spoken Language: A Literature ReviewLanguage and Speech, 1997
- Pitch perception by cochlear implant subjectsThe Journal of the Acoustical Society of America, 1987
- Quantifying the contribution of vision to speech perception in noiseBritish Journal of Audiology, 1987
- Acoustical aspects of contrastive stress in question–answer contextsThe Journal of the Acoustical Society of America, 1985
- Multichannel electrical stimulation of the auditory nerve in man. I. Basic psychophysicsHearing Research, 1983
- Head Movement Correlates of Juncture and Stress at Sentence LevelLanguage and Speech, 1983
- Hearing lips and seeing voicesNature, 1976