INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
Authors:
- Angelika RomanouARAngelika Romanou
- Snegha ASASnegha A
Preprint
Abstract
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistantsProceedings of the National Academy of Sciences, 2024
- Cultural bias and cultural alignment of large language modelsPNAS Nexus, 2024
- Translation Technology in Accessible Health Communication2023
- Gender Bias in Machine TranslationTransactions of the Association for Computational Linguistics, 2021
- Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation2021
- XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation2020
- EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- How Human is Machine Translationese? Comparing Human and Machine Translations of Text and Speech2020
- The State and Fate of Linguistic Diversity and Inclusion in the NLP World2020