ATF-MUSE: A Lightweight U-Net-Based Model With Adaptive Time-Frequency Transformer for Speech Enhancement
Authors:
Open Access
- 23 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 46294-46303
Abstract
Speech enhancement techniques based on deep learning have demonstrated substantial improvements over traditional signal processing methods, particularly under complex non-stationary noise conditions. In this paper, we propose a novel lightweight speech enhancement model, ATF-MUSE, which is designed to more effectively model the complex interactions between temporal- and frequency-domain features, while dynamically adjusting attention weights to enhance adaptability to varying noise environments. Specifically, we introduce an ATF-Transformer module that alternately models dependencies along the temporal and frequency dimensions, thereby capturing both the global spectral patterns and long-term temporal relationships of speech signals. Moreover, we propose an Adaptive Multi-Head Attention (AMHA) mechanism that dynamically generates feature-dependent attention weights, significantly improving the model’s robustness against non-stationary noise. We evaluate the proposed ATF-MUSE model on the widely used VoiceBank-DEMAND dataset, and experimental results demonstrate that our method outperforms several state-of-the-art approaches across multiple standard evaluation metrics, including PESQ, MOS, and STOI.Keywords
Funding Information
- National Science and Technology Major Project (2025ZD0619602)
This publication has 27 references indexed in Scilit:
- Synthetic aperture image enhancement with near-coinciding Nonuniform sampling caseComputers and Electrical Engineering, 2024
- MANNER: Multi-View Attention Network For Noise Erasure2022
- Dual-Branch Attention-In-Attention Transformer for Single-Channel Speech Enhancement2022
- DPT-FSNet: Dual-Path Transformer Based Full-Band and Sub-Band Fusion Network for Speech Enhancement2022
- TSTNN: Two-Stage Transformer Based Neural Network for Speech Enhancement in the Time Domain2021
- Sergan: Speech Enhancement Using Relativistic Generative Adversarial Networks with Gradient Penalty2019
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Speech Enhancement with LSTM Recurrent Neural Networks and its Application to Noise-Robust ASR2015
- Speech enhancement with an adaptive Wiener filterInternational Journal of Speech Technology, 2013
- Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs2002