Putting the “Brain” Back in the Eye-Mind Link: Aligning Eye Movements and Brain Activations During Naturalistic Reading
Open Access
- 4 November 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 407-417
Abstract
Eye movements have long been used to reflect ongoing cognitive processing to develop explanatory and predictive models of mental states and processes. This relationship, deemed the eye-mind link, contains underlying assumptions of the mental processes occurring in the brain, which have rarely been explicitly investigated. We propose a multimodal approach to investigate alignment of eye movements and brain activations (eye-brain alignment) and how it might be predicted by unfolding cognitive processes. We applied this method to a dataset of 76 participants who read long, connected texts while their eye movements and the hemodynamic responses in their brains were tracked using functional near-infrared spectroscopy (fNIRS). We found that reliable eye-brain alignment signals varied based on the participants’ cognitive state during reading. Implications for multimodal modeling of cognitive processes are discussed.Keywords
Funding Information
- National Science Foundation (DRL 1920510)
This publication has 84 references indexed in Scilit:
- Gaze-based predictive models of deep reading comprehensionUser Modeling and User-Adapted Interaction, 2022
- Cortical control of eye movements in natural reading: Evidence from MVPAExperimental Brain Research, 2019
- The validity of the online thought-probing procedure of mind wandering is not threatened by variations of probe rate and probe framingPsychological Research, 2019
- Combined eye tracking and fMRI reveals neural basis of linguistic predictions during sentence comprehensionCortex, 2015
- Neural correlates of impaired emotional face recognition in cerebellar lesionsBrain Research, 2015
- Syntactic processing in the human brain: What we know, what we don’t know, and a suggestion for how to proceedBrain and Language, 2012
- Computational models of eye-movement control during reading: Theories of the “eye–mind” linkCognitive Systems Research, 2006
- Auditory Language Comprehension: An Event-Related fMRI Study on the Processing of Syntactic and Lexical InformationBrain and Language, 2000
- Resolution of syntactic category ambiguities: Eye movements in parsing lexically ambiguous sentencesJournal of Memory and Language, 1987
- What guides a reader's eye movements?Vision Research, 1976