Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition
Preprint
Abstract
Transformers have recently dominated the ASR field. Although able to yield good performance, they involve an autoregressive (AR) decoder to generate tokens one by one, which is computationally inefficient. To speed up inference, non-autoregressive (NAR) methods, e.g. single-step NAR, were designed, to enable parallel generation. However, due to an independence assumption within the output tokens, performance of single-step NAR is inferior to that of AR models, especially with a large-scale corpus. There are two challenges to improving single-step NAR: Firstly to accurately predict the number of output tokens and extract hidden variables; secondly, to enhance modeling of interdependence between output tokens. To tackle both challenges, we propose a fast and accurate parallel transformer, termed Paraformer. This utilizes a continuous integrate-and-fire based predictor to predict the number of tokens and generate hidden variables. A glancing language model (GLM) sampler then generates semantic embeddings to enhance the NAR decoder's ability to model context interdependence. Finally, we design a strategy to generate negative samples for minimum word error rate training to further improve performance. Experiments using the public AISHELL-1, AISHELL-2 benchmark, and an industrial-level 20,000 hour task demonstrate that the proposed Paraformer can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Improving CTC-Based Speech Recognition Via Knowledge Transferring from Pre-Trained Language Models2022
- Non-Autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognition2022
- Improving Non-Autoregressive End-to-End Speech Recognition with Pre-Trained Acoustic and Language Models2022
- Fast End-to-End Speech Recognition Via Non-Autoregressive Models and Cross-Modal Knowledge Transferring From BERTIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Glancing Transformer for Non-Autoregressive Neural Machine Translation2021
- Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict2020
- Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition2020
- Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition2020
- ESPnet: End-to-End Speech Processing Toolkit2018
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement2018