Adaptable Multi-Domain Language Model for Transformer ASR
Authors:
- 6 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7358-7362
Abstract
We propose an adapter based multi-domain Transformer based language model (LM) for Transformer ASR. The model consists of a big size common LM and small size adapters. The model can perform multi-domain adaptation with only the small size adapters and its related layers. The proposed model can reuse the full fine-tuned LM which is fine-tuned using all layers of an original model. The proposed LM can be expanded to new domains by adding about 2% of parameters for a first domain and 13% parameters for after second domain. The proposed model is also effective in reducing the model maintenance cost because it is possible to omit the costly and time-consuming common LM pre-training process. Using proposed adapter based approach, we observed that a general LM with adapter can outperform a dedicated music domain LM in terms of word error rate (WER).Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model2019
- Knowledge Distillation Using Output Errors for Self-attention End-to-end Models2019
- Efficient Parametrization of Multi-domain Deep Neural Networks2018
- Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition2018
- State-of-the-Art Speech Recognition with Sequence-to-Sequence Models2018
- An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model2018
- Deep Contextualized Word Representations2018
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Learning Without Forgetting2016
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989