Linguistic properties and model scale in brain encoding: from small to compressed language models
Preprint
Abstract
Recent work has shown that scaling large language models (LLMs) improves their alignment with human brain activity, yet it remains unclear what drives these gains and which representational properties are responsible. Although larger models often yield better task performance and brain alignment, they are increasingly difficult to analyze mechanistically. This raises a fundamental question: what is the minimal model capacity required to capture brain-relevant representations? To address this question, we systematically investigate how constraining model scale and numerical precision affects brain alignment. We compare full-precision LLMs, small language models (SLMs), and compressed variants (quantized and pruned) by predicting fMRI responses during naturalistic language comprehension. Across model families up to 14B parameters, we find that 3B SLMs achieve brain predictivity indistinguishable from larger LLMs, whereas 1B models degrade substantially, particularly in semantic language regions. Brain alignment is remarkably robust to compression: most quantization and pruning methods preserve neural predictivity, with GPTQ as a consistent exception. Linguistic probing reveals a dissociation between task performance and brain predictivity: compression degrades discourse, syntax, and morphology, yet brain predictivity remains largely unchanged. Overall, brain alignment saturates at modest model scales and is resilient to compression, challenging common assumptions about neural scaling and motivating compact models for brain-aligned language modeling.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Generative language reconstruction from brain recordingsCommunications Biology, 2025
- Phonemic segmentation of narrative speech in human cerebral cortexNature Communications, 2023
- Combining computational controls with natural text reveals aspects of meaning compositionNature Computational Science, 2022
- Proper and common names in the semantic systemBrain Structure and Function, 2022
- Shared computational principles for language processing in humans and deep language modelsNature Neuroscience, 2022
- Brains and algorithms partially converge in natural language processingCommunications Biology, 2022
- The neural architecture of language: Integrative modeling converges on predictive processingProceedings of the National Academy of Sciences, 2021
- Parcellation‐based anatomic model of the semantic networkBrain and Behavior, 2021
- A multi-modal parcellation of human cerebral cortexNature, 2016
- Natural speech reveals the semantic maps that tile human cerebral cortexNature, 2016