Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
- 13 February 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 31 (23299290) , 1075-1086
Abstract
The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharing mechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL ( F ine- and C oarse- Granularity C ontrastive L earning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic information. In addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62171470, 62276153)
- Central Plains Science and Technology Innovation Leading Talent Foundation
This publication has 46 references indexed in Scilit:
- Attention-Passing Models for Robust and Data-Efficient End-to-End Speech TranslationTransactions of the Association for Computational Linguistics, 2019
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation2019
- Pre-training on high-resource speech recognition improves low-resource speech-to-text translation2019
- End-to-End Automatic Speech Translation of Audiobooks2018
- Sequence-to-Sequence Models Can Directly Translate Foreign Speech2017
- Neural Machine Translation of Rare Words with Subword Units2016
- An Attentional Model for Speech Translation Without Transcription2016
- Librispeech: An ASR corpus based on public domain audio books2015
- Statistical Phrase-Based Speech Translation2006
- Speech translation: coupling of recognition and translation1999