Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR
Authors:
- Rami BotrosRBRami Botros
- Anmol GulatiAGAnmol Gulati
- Tara N. SainathTSTara N. Sainath
- Krzysztof ChoromanskiKCKrzysztof Choromanski
- Ruoming PangRPRuoming Pang
- Trevor StrohmanTSTrevor Strohman
- Weiran WangWWWeiran Wang
- Jiahui YuJYJiahui Yu
Preprint
Abstract
Conformer models maintain a large number of internal states, the vast majority of which are associated with self-attention layers. With limited memory bandwidth, reading these from memory at each inference step can slow down inference. In this paper, we design an optimized conformer that is small enough to meet on-device restrictions and has fast inference on TPUs. We explore various ideas to improve the execution speed, including replacing lower conformer blocks with convolution-only blocks, strategically downsizing the architecture, and utilizing an RNNAttention-Performer. Our optimized conformer can be readily incorporated into a cascaded-encoder setting, allowing a second-pass decoder to operate on its output and improve the accuracy whenever more resources are available. Altogether, we find that these optimizations can reduce latency by a factor of 6.8x, and come at a reasonable trade-off in quality. With the cascaded second-pass, we show that the recognition accuracy is completely recoverable. Thus, our proposed encoder can double as a strong standalone encoder in on device, and as the first part of a high-performance ASR pipeline.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Efficient Conformer: Progressive Downsampling and Grouped Attention for Automatic Speech Recognition2021
- An Efficient Streaming Non-Recurrent On-Device End-to-End Model with Improvements to Rare-Word Modeling2021
- A Better and Faster end-to-end Model for Streaming ASR2021
- Finetuning Pretrained Transformers into RNNs2021
- Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss2020
- A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency2020
- Generation of Large-Scale Simulated Utterances in Virtual Rooms to Train Deep-Neural Networks for Far-Field Speech Recognition in Google Home2017
- Joint CTC-attention based end-to-end speech recognition using multi-task learning2017
- Large scale deep neural network acoustic modeling with semi-supervised training data for YouTube video transcription2013