Brain-Informed Fine-Tuning for Improved Multilingual Understanding in Language Models
Preprint
Abstract
Recent studies have demonstrated that fine-tuning language models with brain data can improve their semantic understanding, although these findings have so far been limited to English. Interestingly, similar to the shared multilingual embedding space of pretrained multilingual language models, human studies provide strong evidence for a shared semantic system in bilingual individuals. Here, we investigate whether fine-tuning language models with bilingual brain data changes model representations in a way that improves them across multiple languages. To test this, we fine-tune monolingual and multilingual language models using brain activity recorded while bilingual participants read stories in English and Chinese. We then evaluate how well these representations generalize to the bilingual participants’ first language, their second language, and several other languages that the participants are not fluent in. We assess the fine-tuned language models on brain encoding performance and downstream NLP tasks. Our results show that bilingual brain-informed fine-tuned language models outperform their vanilla (pretrained) counterparts in both brain encoding performance and most downstream NLP tasks across multiple languages. These findings suggest that brain-informed fine-tuning improves multilingual understanding in language models, offering a bridge between cognitive neuroscience and NLP research. We make our code publicly available. 2Keywords
Funding Information
- German Federal Ministry of Education and Research (01GQ1906)
- European Research Council (101042567)
This publication has 28 references indexed in Scilit:
- Multilingual Computational Models Capture a Shared Meaning Component in Brain Responses across 21 LanguagesbioRxiv, 2025
- The cortical representation of language timescales is shared between reading and listeningCommunications Biology, 2024
- On Bilinguals and Bilingualism2024
- Shared computational principles for language processing in humans and deep language modelsNature Neuroscience, 2022
- Brains and algorithms partially converge in natural language processingCommunications Biology, 2022
- The neural architecture of language: Integrative modeling converges on predictive processingbioRxiv, 2020
- Incorporating Context into Language Encoding Models for fMRIbioRxiv, 2018
- Natural speech reveals the semantic maps that tile human cerebral cortexNature, 2016
- Encoding and decoding in fMRINeuroImage, 2011
- Bilingual Semantic and Conceptual Representation2009