Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
Authors:
- Hao ZhangHZHao Zhang
- Nianwen SiNSNianwen Si
- Yaqi ChenYCYaqi Chen
- Wenlin ZhangWZWenlin Zhang
- Xukui YangXYXukui Yang
- Dan QuDQDan Qu
- Wei-Qiang ZhangWZWei-Qiang Zhang
Preprint
Abstract
The end-to-end speech translation (E2E-ST) model has gradually become a mainstream paradigm due to its low latency and less error propagation. However, it is non-trivial to train such a model well due to the task complexity and data scarcity. The speech-and-text modality differences result in the E2E-ST model performance usually inferior to the corresponding machine translation (MT) model. Based on the above observation, existing methods often use sharingmechanisms to carry out implicit knowledge transfer by imposing various constraints. However, the final model often performs worse on the MT task than the MT model trained alone, which means that the knowledge transfer ability of this method is also limited. To deal with these problems, we propose the FCCL (Fine- and Coarse- Granularity Contrastive Learning) approach for E2E-ST, which makes explicit knowledge transfer through cross-modal multi-grained contrastive learning. A key ingredient of our approach is applying contrastive learning at both sentence- and frame-level to give the comprehensive guide for extracting speech representations containing rich semantic information.In addition, we adopt a simple whitening method to alleviate the representation degeneration in the MT model, which adversely affects contrast learning. Experiments on the MuST-C benchmark show that our proposed approach significantly outperforms the state-of-the-art E2E-ST baselines on all eight language pairs. Further analysis indicates that FCCL can free up its capacity from learning grammatical structure information and force more layers to learn semantic information.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Cross-Lingual Self-training to Learn Multilingual Representation for Low-Resource Speech RecognitionCircuits, Systems, and Signal Processing, 2022
- Regularizing End-to-End Speech Translation with Triangular Decomposition AgreementProceedings of the AAAI Conference on Artificial Intelligence, 2022
- DropDim: A Regularization Method for Transformer NetworksIEEE Signal Processing Letters, 2022
- Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task2021
- ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer2021
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation2020
- Adaptive Feature Selection for End-to-End Speech Translation2020
- Curriculum Pre-training for End-to-End Speech Translation2020
- Attention-Passing Models for Robust and Data-Efficient End-to-End Speech TranslationTransactions of the Association for Computational Linguistics, 2019