HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism
Preprint
Abstract
SqueezeFormer has recently shown impressive performance in automatic speech recognition (ASR). However, its inference speed suffers from the quadratic complexity of softmax-attention (SA). In addition, limited by the large convolution kernel size, the local modeling ability of SqueezeFormer is insufficient. In this paper, we propose a novel method HybridFormer to improve SqueezeFormer in a fast and efficient way. Specifically, we first incorporate linear attention (LA) and propose a hybrid LASA paradigm to increase the model's inference speed. Second, a hybrid neural architecture search (NAS) guided structural re-parameterization (SRep) mechanism, termed NSR, is proposed to enhance the ability of the model to extract local interactions. Extensive experiments conducted on the LibriSpeech dataset demonstrate that our proposed HybridFormer can achieve a 9.1% relative word error rate (WER) reduction over SqueezeFormer on the test-other dataset. Furthermore, when input speech is 30s, the HybridFormer can improve the model's inference speed up to 18%. Our source code is available online.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition2022
- Conformer: Convolution-augmented Transformer for Speech Recognition2020
- Jasper: An End-to-End Convolutional Neural Acoustic Model2019
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition2019
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018