Explaining neural activity in human listeners with deep learning via natural language processing of narrative text
Open Access
- 25 October 2022
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 12 (1) , 1-9
Abstract
Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction.Keywords
This publication has 38 references indexed in Scilit:
- Salience and Attention in Surprisal-Based Accounts of Language ProcessingFrontiers in Psychology, 2016
- What do we mean by prediction in language comprehension?Language, Cognition and Neuroscience, 2015
- Prediction During Natural Language ComprehensionCerebral Cortex, 2015
- Commentary on “The role of domain-general cognitive control in language comprehension†by FedorenkoFrontiers in Psychology, 2014
- The effect of word predictability on reading time is logarithmicCognition, 2013
- Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated StoryThe Journal of Neuroscience, 2011
- The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviourTrends in Cognitive Sciences, 2010
- Data from eye-tracking corpora as evidence for theories of syntactic processing complexityCognition, 2008
- Segmented Discourse Representation Theory: Dynamic Semantics With Discourse Structure2008
- The cortical organization of speech processingNature Reviews Neuroscience, 2007