Kaja Rosa Benz

While eye movements have been shown to track the speech envelope, it is unknown whether this reflects a hard-wired mechanism or one shaped by (lifetime) audiovisual experience. Further, questions remain about whether ocular tracking is modulated by speech intelligibility and which brain regions drive these synchronized eye movements. Here, we investigate ocular speech tracking in blindfolded early blind, late blind, and sighted individuals using magnetoencephalography (MEG) and source-reconstructed oculomotor signals while participants listened to narrative speech of varying intelligibility. We found that oculomotor activity tracks acoustic speech features and, unlike neural speech tracking, is not modulated by intelligibility. Interestingly, we found effects reflected in two frequency-specific components: a low-frequency (∼1 Hz) effect present across all groups, indicating that visual experience is not required, and a high-frequency (∼6 Hz) effect reduced in early- and late-blind individuals. Moreover, this finding is not driven by cerebro-ocular connectivity, as late-blind individuals exhibit stronger connectivity between the eyes and the left temporal cortices without a corresponding increase in ocular tracking. In conclusion, ocular speech tracking seems to respond selectively to acoustic but not to intelligibility features of speech, and it does not require visual experience to develop. It may thus represent a hard-wired oculomotor mechanism within the oculo-cerebral network involved in speech processing. Significance Statement: Eye movements provide a unique window into the interaction between auditory and visual systems. By studying early blind, late blind, and sighted individuals, we demonstrate that speech-related eye movements arise from at least two distinct mechanisms: a low-frequency component that occurs independently of (lifetime) visual experience and is linked to processing of acoustic speech features, and a high-frequency component shaped by prior visual exposure. Importantly, speech intelligibility - unlike its impact on neural measures - does not modulate these ocular responses. This dissociation suggests that eye movements reflect mechanisms of spoken language processing that are independent of intelligibility, thereby revealing novel pathways of auditory-motor coupling and broadening our understanding of sensory integration in the absence of vision.

Behavioral and neuroscientific studies have shown that watching a speaker's lip movements aids speech comprehension. Intriguingly, even when videos of speakers are presented silently, various cortical regions track auditory features, such as the envelope. Recently, we demonstrated that eye movements track low-level acoustic information when attentively listening to speech. In this study, we investigated whether ocular speech tracking occurs during visual speech and how it influences cortical silent speech tracking. Furthermore, we compared data from hearing individuals, congenitally deaf individuals, and those who became deaf or hard of hearing (DHH) later in life to assess how audiovisual listening experience and auditory deprivation (early vs late onset) affect neural and ocular speech tracking during silent lip-reading. Using magnetoencephalography (MEG), we examined ocular and neural speech tracking of 75 participants observing silent videos of a speaker played forward and backward. Our main finding is a clear ocular unheard speech tracking effect with dominance of <1 Hz, which was not present for lip movements. Similarly, we observed an ≤ 1.3 Hz effect of neural unheard speech tracking in temporal regions for hearing participants. Importantly, neural tracking was not directly linked to ocular tracking. Strikingly, across listening groups, deaf participants with auditory experience showed higher ocular speech tracking than hearing participants, while no ocular speech tracking effect was revealed for congenitally deaf participants in a very small sample. This study extends previous work by demonstrating the involvement of eye movements in speech processing, even in the absence of acoustic input.

Behavioral and neuroscientific studies have shown that watching a speaker’s lip movements aids speech comprehension. Intriguingly, even when videos of speakers are presented silently, various cortical regions track auditory features, such as the envelope. Recently, we demonstrated that eye movements track low-level acoustic information when attentively listening to speech. In this study we investigated whether ocular speech tracking occurs during visual speech and how it influences cortical silent speech tracking. Furthermore, we compared the data of hearing individuals with congenitally deaf individuals, and those with acquired deafness or hearing loss (DHH; Deaf or hard of hearing) to assess how auditory deprivation (early vs. late onset) affects neural and ocular speech tracking during silent lip-reading. Using magnetoencephalography (MEG), we examined ocular and neural speech tracking of 75 participants observing silent videos of a speaker played forward and backward. Our main finding is a clear ocular unheard speech tracking effect with a dominance <1 Hz, which was not present for the lip movements. Similarly, we observed a <=1.3 Hz effect of neural unheard speech tracking in temporal regions for hearing participants. Importantly, neural tracking was not directly linked to ocular tracking in this study. Strikingly, across different listening groups, deaf participants with auditory listening experience showed higher ocular speech tracking than hearing participants, while no ocular speech tracking effect was revealed for congenitally deaf participants in a very small sample. This study extends our previous work by demonstrating the involvement of eye movements in speech processing, even in the absence of acoustic input.

Regularities in our surroundings lead to predictions about upcoming events. Previous research has shown, that omitted sounds during otherwise regular tone sequences elicit frequency-specific neural activity related to the upcoming but omitted tone. We tested whether this neural response is depending on the unpredictability of the omission. Therefore we recorded MEG data while participants listened to ordered or random tone sequences with omissions occurring either ordered or randomly. Using multivariate pattern analysis shows that the frequency-specific neural pattern during omission within ordered tone sequences occurs independent of the regularity of the omissions suggesting that the auditory predictions based on sensory experiences are not immediately updated by violations of those expectations.