A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
Preprint
Abstract
Attention-based sequence-to-sequence modeling provides a powerful and elegant solution for applications that need to map one sequence to a different sequence. Its success heavily relies on the availability of large amounts of training data. This presents a challenge for speech applications where labelled speech data is very expensive to obtain, such as automatic speech recognition (ASR) and speech translation (ST). In this study, we propose a general multi-task learning framework to leverage text data for ASR and ST tasks. Two auxiliary tasks, a denoising autoencoder task and machine translation task, are proposed to be co-trained with ASR and ST tasks respectively. We demonstrate that representing text input as phoneme sequences can reduce the difference between speech and text inputs, and enhance the knowledge transfer from text corpora to the speech to text tasks. Our experiments show that the proposed method achieves a relative 10~15% word error rate reduction on the English Librispeech task compared with our baseline, and improves the speech translation quality on the MuST-C tasks by 3.6~9.2 BLEU.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Leveraging Unpaired Text Data for Training End-To-End Speech-to-Intent Systems2020
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq2020
- Phone Features Improve Speech Translation2020
- ESPnet-ST: All-in-One Speech Translation Toolkit2020
- Speech Recognition with Augmented Synthesized Speech2019
- A Comparative Study on End-to-End Speech to Text Translation2019
- Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation2019
- A study on data augmentation of reverberant speech for robust speech recognition2017
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Librispeech: An ASR corpus based on public domain audio books2015