On Backbones and Training Regimes for Dense Retrieval in African Languages
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2564-2568
Abstract
The effectiveness of dense retrieval models trained with multilingual language models as backbones has been demonstrated in multilingual and cross-lingual information retrieval contexts. The optimal choice of a backbone model for a given retrieval task is dependent on the target retrieval domain as well as the pre-training domain of available language models and their generalization capabilities, the availability of relevance judgements, etc. In this work, we study the impact of these factors on retrieval effectiveness for African languages using three multilingual benchmark datasets: Mr. TyDi, MIRACL, and the newly released CIRAL dataset. We compare the effectiveness of mBERT as a backbone for dense retrieval models against multilingual language models such as AfriBERTa and AfroXLMR, which are specialized for African languages. Furthermore, we examine the impact of different training regimes on the effectiveness of dense retrieval in different domains for African languages. Our findings show that the pre-training domain of the backbone LM plays a huge role in retrieval effectiveness, especially in the absence of retrieval training data. Code artifacts are available at https://github.com/castorini/afridpr_backbones.Keywords
This publication has 13 references indexed in Scilit:
- CIRAL at FIRE 2023: Cross-Lingual Information Retrieval for African Languages2023
- Toward Best Practices for Training Multilingual Dense Retrieval ModelsACM Transactions on Information Systems, 2023
- MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse LanguagesTransactions of the Association for Computational Linguistics, 2023
- MasakhaNEWS: News Topic Classification for African languages2023
- Better Quality Pre-training Data and T5 Models for African Languages2023
- Cross-lingual Open-Retrieval Question Answering for African Languages2023
- MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition2022
- ColBERT2020
- Dense Passage Retrieval for Open-Domain Question Answering2020
- Unsupervised Cross-lingual Representation Learning at Scale2020