DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment
Authors:
- Abderrahmane IssamAIAbderrahmane Issam
- Jan ScholtesJSJan Scholtes
- Gerasimos SpanakisGSGerasimos Spanakis
Preprint
Abstract
End-to-End Speech Translation (E2E-ST) is the task of translating source speech directly into target text bypassing the intermediate transcription step. The representation discrepancy between the speech and text modalities has motivated research on what is known as bridging the modality gap. State-of-the-art methods addressed this by aligning speech and text representations on the word or token level. Unfortunately, this requires an alignment tool that is not available for all languages. Although this issue has been addressed by aligning speech and text embeddings using nearest-neighbor similarity search, it does not lead to accurate alignments. In this work, we adapt Dynamic Time Warping (DTW) for aligning speech and text embeddings during training. Our experiments demonstrate the effectiveness of our method in bridging the modality gap in E2E-ST. Compared to previous work, our method produces more accurate alignments and achieves comparable E2E-ST results while being significantly faster. Furthermore, our method outperforms previous work in low resource settings on 5 out of 6 language directions.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Soft Alignment of Modality Space for End-to-End Speech Translation2024
- Improving Speech Translation by Cross-Modal Multi-Grained Contrastive LearningIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Cross-modal Contrastive Learning for Speech Translation2022
- STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Learning Shared Semantic Space for Speech-to-Text Translation2021
- Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task2021
- CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition2020
- On the Hidden Markov Model and Dynamic Time Warping for Speech Recognition-A Unified ViewAT&T Bell Laboratories Technical Journal, 1984
- Dynamic programming algorithm optimization for spoken word recognitionIEEE Transactions on Acoustics, Speech, and Signal Processing, 1978