Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation
Authors:
- Yuhao ZhangYZYuhao Zhang
- Tong XiaoTXTong Xiao
- Zhengtao YuZYZhengtao Yu
- Jingbo ZhuJZJingbo Zhu
Preprint
Abstract
The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using $n$-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Textless Direct Speech-to-Speech Translation with Discrete Speech Representation2023
- UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units2023
- Textless Speech-to-Speech Translation on Real Data2022
- Direct Speech-to-Speech Translation With Discrete Units2022
- UWSpeech: Speech to Speech Translation for Unwritten LanguagesProceedings of the AAAI Conference on Artificial Intelligence, 2021
- VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation2021
- Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders2021
- Speech-to-Speech Translation Between Untranscribed Unknown Languages2019
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Connectionist temporal classification2006