Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training
Preprint
Abstract
Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models’ ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity—a measure of next-word prediction performance—is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models’ predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.Keywords
All Related Versions
- Published version: Neurobiology of Language, 5 (1), 43.
This publication has 61 references indexed in Scilit:
- The effect of word predictability on reading time is logarithmicCognition, 2013
- Power failure: why small sample size undermines the reliability of neuroscienceNature Reviews Neuroscience, 2013
- False-Positive PsychologyPsychological Science, 2011
- Functional specificity for high-level linguistic processing in the human brainProceedings of the National Academy of Sciences, 2011
- Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated StoryThe Journal of Neuroscience, 2011
- Cortical representation of the constituent structure of sentencesProceedings of the National Academy of Sciences, 2011
- New Method for fMRI Investigations of Language: Defining ROIs Functionally in Individual SubjectsJournal of Neurophysiology, 2010
- Language acquisition in the absence of explicit negative evidence: how important is starting small?Cognition, 1999
- American parenting of language-learning children: Persisting differences in family-child interactions observed in natural home environments.Developmental Psychology, 1992
- Perplexity—a measure of the difficulty of speech recognition tasksThe Journal of the Acoustical Society of America, 1977