XTREME-S: Evaluating Cross-lingual Speech Representations
Preprint
Abstract
We introduce XTREME-S, a new benchmark to evaluate universal cross-lingual speech representations in many languages. XTREME-S covers four task families: speech recognition, classification, speech-to-text translation and retrieval. Covering 102 languages from 10+ language families, 3 different domains and 4 task families, XTREME-S aims to simplify multilingual speech representation evaluation, as well as catalyze research in "universal" speech representation learning. This paper describes the new benchmark and establishes the first speech-only and speech-text baselines using XLS-R and mSLAM on all downstream tasks. We motivate the design choices and detail how to use the benchmark. Datasets and fine-tuning scripts are made easily accessible at https://hf.co/datasets/google/xtreme_s.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training2021
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models2021
- VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation2021
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesTransactions of the Association for Computational Linguistics, 2020
- MLQA: Evaluating Cross-lingual Extractive Question Answering2020
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019
- Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion2019
- Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks2018
- The 2016 BBN Georgian telephone speech keyword spotting system2017