Cross-Modal Transfer Learning for Multilingual Speech-to-Text Translation
Authors:
- Chau TranCTChau Tran
- Changhan WangCWChanghan Wang
- Yuqing TangYTYuqing Tang
- Yun TangYTYun Tang
- Juan PinoJPJuan Pino
- Xian LiXLXian Li
Preprint
Abstract
We propose an effective approach to utilize pretrained speech and text models to perform speech-to-text translation (ST). Our recipe to achieve cross-modal and cross-lingual transfer learning (XMTL) is simple and generalizable: using an adaptor module to bridge the modules pretrained in different modalities, and an efficient finetuning step which leverages the knowledge from pretrained modules yet making it work on a drastically different downstream task. With this approach, we built a multilingual speech-to-text translation model with pretrained audio encoder (wav2vec) and multilingual text decoder (mBART), which achieves new state-of-the-art on CoVoST 2 ST benchmark [1] for English into 15 languages as well as 6 Romance languages into English with on average +2.8 BLEU and +3.9 BLEU, respectively. On low-resource languages (with less than 10 hours training data), our approach significantly improves the quality of speech-to-text translation with +9.0 BLEU on Portuguese-English and +5.2 BLEU on Dutch-English.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- End-end Speech-to-Text Translation with Modality Agnostic Meta-Learning2020
- Unsupervised Pretraining Transfers Well Across Languages2020
- Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech TranslationProceedings of the AAAI Conference on Artificial Intelligence, 2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Curriculum Pre-training for End-to-End Speech Translation2020
- Improved Speech Representations with Multi-Target Autoregressive Predictive Coding2020
- A Comparative Study on End-to-End Speech to Text Translation2019
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation2019
- End-to-End Automatic Speech Translation of Audiobooks2018