The effects of temporal cues, point-light displays, and faces on speech identification and listening effort
Open Access
- 29 November 2023
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 18 (11) , e0290826
Abstract
Among the most robust findings in speech research is that the presence of a talking face improves the intelligibility of spoken language. Talking faces supplement the auditory signal by providing fine phonetic cues based on the placement of the articulators, as well as temporal cues to when speech is occurring. In this study, we varied the amount of information contained in the visual signal, ranging from temporal information alone to a natural talking face. Participants were presented with spoken sentences in energetic or informational masking in four different visual conditions: audio-only, a modulating circle providing temporal cues to salient features of the speech, a digitally rendered point-light display showing lip movement, and a natural talking face. We assessed both sentence identification accuracy and self-reported listening effort. Audiovisual benefit for intelligibility was observed for the natural face in both informational and energetic masking, but the digitally rendered point-light display only provided benefit in energetic masking. Intelligibility for speech accompanied by the modulating circle did not differ from the audio-only conditions in either masker type. Thus, the temporal cues used here were insufficient to improve speech intelligibility in noise, but some types of digital point-light displays may contain enough phonetic detail to produce modest improvements in speech identification in noise.Keywords
Funding Information
- Carleton College and National Institute on Deafness and Other Communication Disorders (R15-DC018114)
This publication has 45 references indexed in Scilit:
- Random effects structure for confirmatory hypothesis testing: Keep it maximalJournal of Memory and Language, 2013
- Cross-Modal Enhancement of Speech Detection in Young and Older Adults: Does Signal Content Matter?Ear & Hearing, 2011
- Similarity and familiarity: Second language sentence recognition in first- and second-language multi-talker babbleSpeech Communication, 2010
- Auditory and Visual Lexical Neighborhoods in Audiovisual Speech PerceptionTrends in Amplification, 2007
- Seeing to hear better: evidence for early audio-visual interactions in speech identificationCognition, 2004
- Measures of auditory–visual integration in nonsense syllables and sentencesThe Journal of the Acoustical Society of America, 1998
- Effects of fluctuating noise and interfering speech on the speech-reception threshold for impaired and normal hearingThe Journal of the Acoustical Society of America, 1990
- Use of Visual Information for Phonetic PerceptionPhonetica, 1979
- The Bkb (Bamford-Kowal-Bench) Sentence Lists for Partially-Hearing ChildrenBritish Journal of Audiology, 1979
- Visual Contribution to Speech Intelligibility in NoiseThe Journal of the Acoustical Society of America, 1954