The neural architecture of language: Integrative modeling converges on predictive processing
Preprint
Abstract
The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a first systematic study taking this approach to higher-level cognition: human language processing, our species’ signature cognitive skill. We find that the most powerful ‘transformer’ models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (fMRI, ECoG). Models’ neural fits (‘brain score’) and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain. Significance: Language is a quintessentially human ability. Research has long probed the functional architecture of language processing in the mind and brain using diverse brain imaging, behavioral, and computational modeling approaches. However, adequate neurally mechanistic accounts of how meaning might be extracted from language are sorely lacking. Here, we report an important first step toward addressing this gap by connecting recent artificial neural networks from machine learning to human recordings during language processing. We find that the most powerful models predict neural and behavioral responses across different datasets up to noise levels. Models that perform better at predicting the next word in a sequence also better predict brain measurements – providing computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.Keywords
All Related Versions
- Published version: Proceedings of the National Academy of Sciences, 118 (45).
This publication has 144 references indexed in Scilit:
- Rational integration of noisy evidence and prior semantic expectations in sentence interpretationProceedings of the National Academy of Sciences, 2013
- Lexical and syntactic representations in the brain: An fMRI investigation with multi-voxel pattern analysesNeuropsychologia, 2012
- Functional specificity for high-level linguistic processing in the human brainProceedings of the National Academy of Sciences, 2011
- Cortical representation of the constituent structure of sentencesProceedings of the National Academy of Sciences, 2011
- New Method for fMRI Investigations of Language: Defining ROIs Functionally in Individual SubjectsJournal of Neurophysiology, 2010
- Data from eye-tracking corpora as evidence for theories of syntactic processing complexityCognition, 2008
- Computational principles of working memory in sentence comprehensionTrends in Cognitive Sciences, 2006
- Toward a Connectionist Model of Recursion in Human Linguistic PerformanceCognitive Science, 1999
- A Probabilistic Model of Lexical and Syntactic Access and DisambiguationCognitive Science, 1996
- Neocognitron: A hierarchical neural network capable of visual pattern recognitionNeural Networks, 1988