Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Tasks
Preprint
Abstract
The Transformer architecture has proven to be highly effective for Automatic Speech Recognition (ASR) tasks, becoming a foundational component for a plethora of research in the domain. Historically, many approaches have leaned on fixed-length attention windows, which becomes problematic for varied speech samples in duration and complexity, leading to data over-smoothing and neglect of essential long-term connectivity. Addressing this limitation, we introduce Echo-MSA, a nimble module equipped with a variable-length attention mechanism that accommodates a range of speech sample complexities and durations. This module offers the flexibility to extract speech features across various granularities, spanning from frames and phonemes to words and discourse. The proposed design captures the variable length feature of speech and addresses the limitations of fixed-length attention. Our evaluation leverages a parallel attention architecture complemented by a dynamic gating mechanism that amalgamates traditional attention with the Echo-MSA module output. Empirical evidence from our study reveals that integrating Echo-MSA into the primary model's training regime significantly enhances the word error rate (WER) performance, all while preserving the intrinsic stability of the original model.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020
- Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss2020
- Large-Scale Unsupervised Pre-Training for End-to-End Spoken Language Understanding2020
- Neural Source-Filter Waveform Models for Statistical Parametric Speech SynthesisIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019
- Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition2018
- A Study of All-Convolutional Encoders for Connectionist Temporal Classification2018
- Focal Loss for Dense Object Detection2017
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012