A bilingual benchmark for evaluating large language models
Open Access
- 29 February 2024
- journal article
- research article
- Published by PeerJ in PeerJ Computer Science
- Vol. 10, e1893
Abstract
This work introduces a new benchmark for the bilingual evaluation of large language models (LLMs) in English and Arabic. While LLMs have transformed various fields, their evaluation in Arabic remains limited. This work addresses this gap by proposing a novel evaluation method for LLMs in both Arabic and English, allowing for a direct comparison between the performance of the two languages. We build a new evaluation dataset based on the General Aptitude Test (GAT), a standardized test widely used for university admissions in the Arab world, that we utilize to measure the linguistic capabilities of LLMs. We conduct several experiments to examine the linguistic capabilities of ChatGPT and quantify how much better it is at English than Arabic. We also examine the effect of changing task descriptions from Arabic to English and vice-versa. In addition to that, we find that fastText can surpass ChatGPT in finding Arabic word analogies. We conclude by showing that GPT-4 Arabic linguistic capabilities are much better than ChatGPT’s Arabic capabilities and are close to ChatGPT’s English capabilities.Keywords
This publication has 15 references indexed in Scilit:
- GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP2023
- ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning2023
- GPT-4 Passes the Bar ExamSSRN Electronic Journal, 2023
- AraT5: Text-to-Text Transformers for Arabic Language Generation2022
- ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic2021
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- CamemBERT: a Tasty French Language Model2020
- Contextual Embeddings: When Are They Worth It?2020
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Glove: Global Vectors for Word Representation2014