DeLiang Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
DeLiang Wang has a total of 101 co-authors. Here are the top co-authors by number of shared publications:
- Ke Tan
Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA
11 shared publications - Ashutosh Pandey
Department of Computer Science and Engineering The Ohio State University USA
9 shared publications - Zhong-Qiu Wang
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
8 shared publications - Heming Wang
Department of Computer Science and Engineering, The Ohio State University 1 , Columbus, Ohio 43210
7 shared publications - Yuzhou Liu
Department of Computer Science and Engineering, The Ohio State University , Columbus, Ohio 43210, USA
6 shared publications Guoning Hu
5 shared publications
Latest Publications
- 1 July 2025
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 158 (1) , 790-800
Robust frame-level speaker localization guided by multi-channel speech enhancement and inter-channel phase-difference losses
In the presence of room reverberation and background noise, the performance of frame-level speaker localization is severely limited. To address this challenge, this study performs multi-channel speech enhancement based on complex spectral mapping (CSM), followed by direction-of-arrival (DOA) estimation using weighted generalized cross-correlation with phase transform (GCC-PHAT). The proposed approach differs from prevailing deep learning methods that operate on multi-channel inputs directly for speaker localization. This study initially investigates multi-input single-output (MISO) based speech enhancement models with two loss functions and then extends to multi-input multi-output (MIMO) based modeling for conceptual and computational efficiency. The results demonstrate that the phase estimates obtained from CSM models are reliable for frame-level DOA estimation and MIMO systems outperform MISO systems. In addition, the study proposes new multi-channel loss functions for MIMO systems that incorporate phase differences in order to better preserve inter-channel phase relations, which is key to accurate sound localization. Systematic evaluations with multiple microphone array geometries using both simulated and recorded room impulse responses, as well as real recordings, demonstrate that the proposed model yields excellent frame-level speaker localization results in reverberant and noisy environments and outperforms related methods by a large margin, even surpassing their utterance-level results.
- 1 July 2025
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 158 (1) , 790-800
Robust frame-level speaker localization guided by multi-channel speech enhancement and inter-channel phase-difference losses
In the presence of room reverberation and background noise, the performance of frame-level speaker localization is severely limited. To address this challenge, this study performs multi-channel speech enhancement based on complex spectral mapping (CSM), followed by direction-of-arrival (DOA) estimation using weighted generalized cross-correlation with phase transform (GCC-PHAT). The proposed approach differs from prevailing deep learning methods that operate on multi-channel inputs directly for speaker localization. This study initially investigates multi-input single-output (MISO) based speech enhancement models with two loss functions and then extends to multi-input multi-output (MIMO) based modeling for conceptual and computational efficiency. The results demonstrate that the phase estimates obtained from CSM models are reliable for frame-level DOA estimation and MIMO systems outperform MISO systems. In addition, the study proposes new multi-channel loss functions for MIMO systems that incorporate phase differences in order to better preserve inter-channel phase relations, which is key to accurate sound localization. Systematic evaluations with multiple microphone array geometries using both simulated and recorded room impulse responses, as well as real recordings, demonstrate that the proposed model yields excellent frame-level speaker localization results in reverberant and noisy environments and outperforms related methods by a large margin, even surpassing their utterance-level results.
- 1 July 2025
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 158 (1) , 380-390
Perceptual effects of reducing algorithmic latency on deep-learning based noise reduction
Low latency is an essential requirement for noise reduction in real-world devices such as hearing aids and cochlear implants. Reducing the algorithmic latency of a deep neural network charged with noise reduction allows additional time for other processing. However, a larger analysis window may be advantageous to the performance of the network. This trade-off is currently examined with regard to human speech-intelligibility performance. The algorithmic latency of the attentive recurrent network (ARN) was modified by reducing the size of the analysis time frame. The ARN model was talker, noise, and recording-channel independent, and fully causal. Listeners with hearing loss and with normal hearing heard sentences in babble at various signal-to-noise ratios. Large increases in intelligibility were observed as a result of noise reduction, especially for the listeners with hearing loss and at less favorable signal-to-noise ratios. Slightly larger objective measures of network performance were observed at larger latencies. But more critically, human performance was essentially unchanged as algorithmic latency was reduced from 20 to 10 or 5 ms. These results are discussed in the context of overall design and implementation of deep-learning based noise reduction, and information on latency requirements for human listeners is summarized.
- 1 July 2025
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 158 (1) , 380-390
Perceptual effects of reducing algorithmic latency on deep-learning based noise reduction
Low latency is an essential requirement for noise reduction in real-world devices such as hearing aids and cochlear implants. Reducing the algorithmic latency of a deep neural network charged with noise reduction allows additional time for other processing. However, a larger analysis window may be advantageous to the performance of the network. This trade-off is currently examined with regard to human speech-intelligibility performance. The algorithmic latency of the attentive recurrent network (ARN) was modified by reducing the size of the analysis time frame. The ARN model was talker, noise, and recording-channel independent, and fully causal. Listeners with hearing loss and with normal hearing heard sentences in babble at various signal-to-noise ratios. Large increases in intelligibility were observed as a result of noise reduction, especially for the listeners with hearing loss and at less favorable signal-to-noise ratios. Slightly larger objective measures of network performance were observed at larger latencies. But more critically, human performance was essentially unchanged as algorithmic latency was reduced from 20 to 10 or 5 ms. These results are discussed in the context of overall design and implementation of deep-learning based noise reduction, and information on latency requirements for human listeners is summarized.
- 7 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 19 (4) , 685-694
AV-CrossNet: An Audiovisual Complex Spectral Mapping Network for Speech Separation by Leveraging Narrow- and Cross-Band Modeling
Adding visual cues to audio-based speech separation can improve separation performance. This paper introduces AV-CrossNet, an audiovisual (AV) system for speech enhancement, target speaker extraction, and multi-talker speaker separation. AV-CrossNet is extended from the TF-CrossNet architecture, which is a recently proposed network that performs complex spectral mapping for speech separation by leveraging global attention and positional encoding. To effectively utilize visual cues, the proposed system incorporates pre-extracted visual embeddings and employs a visual encoder comprising temporal convolutional layers. Audio and visual features are fused in an early fusion layer before feeding to AV-CrossNet blocks. We evaluate AV-CrossNet on multiple datasets, including LRS, VoxCeleb, TCD-TIMIT, and COG-MHEAR challenge, in terms of the performance metrics of PESQ, STOI, SNR and SDR. Evaluation results demonstrate that AV-CrossNet advances the state-of-the-art performance in all audiovisual tasks, even on untrained and mismatched datasets.
- 7 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 19 (4) , 685-694
AV-CrossNet: An Audiovisual Complex Spectral Mapping Network for Speech Separation by Leveraging Narrow- and Cross-Band Modeling
Adding visual cues to audio-based speech separation can improve separation performance. This paper introduces AV-CrossNet, an audiovisual (AV) system for speech enhancement, target speaker extraction, and multi-talker speaker separation. AV-CrossNet is extended from the TF-CrossNet architecture, which is a recently proposed network that performs complex spectral mapping for speech separation by leveraging global attention and positional encoding. To effectively utilize visual cues, the proposed system incorporates pre-extracted visual embeddings and employs a visual encoder comprising temporal convolutional layers. Audio and visual features are fused in an early fusion layer before feeding to AV-CrossNet blocks. We evaluate AV-CrossNet on multiple datasets, including LRS, VoxCeleb, TCD-TIMIT, and COG-MHEAR challenge, in terms of the performance metrics of PESQ, STOI, SNR and SDR. Evaluation results demonstrate that AV-CrossNet advances the state-of-the-art performance in all audiovisual tasks, even on untrained and mismatched datasets.
- 6 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 4999-5009
TF-CrossNet: Leveraging Global, Cross-Band, Narrow-Band, and Positional Encoding for Single- and Multi-Channel Speaker Separation
We introduce TF-CrossNet, a complex spectral mapping approach to speaker separation and enhancement in reverberant and noisy conditions. The proposed architecture comprises an encoder layer, a global multi-head self-attention module, a cross-band module, a narrow-band module, and an output layer. TF-CrossNet captures global, cross-band, and narrow-band correlations in the time-frequency domain. To address performance degradation in long utterances, we introduce a random chunk positional encoding. Experimental results on multiple datasets demonstrate the effectiveness and robustness of TF-CrossNet, achieving state-of-the-art performance in tasks including reverberant and noisy-reverberant speaker separation. Furthermore, TF-CrossNet exhibits faster and more stable training in comparison to recent baselines. Additionally, TF-CrossNet's high performance extends to multi-microphone conditions, demonstrating its versatility in various acoustic scenarios.
- 6 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 4999-5009
TF-CrossNet: Leveraging Global, Cross-Band, Narrow-Band, and Positional Encoding for Single- and Multi-Channel Speaker Separation
We introduce TF-CrossNet, a complex spectral mapping approach to speaker separation and enhancement in reverberant and noisy conditions. The proposed architecture comprises an encoder layer, a global multi-head self-attention module, a cross-band module, a narrow-band module, and an output layer. TF-CrossNet captures global, cross-band, and narrow-band correlations in the time-frequency domain. To address performance degradation in long utterances, we introduce a random chunk positional encoding. Experimental results on multiple datasets demonstrate the effectiveness and robustness of TF-CrossNet, achieving state-of-the-art performance in tasks including reverberant and noisy-reverberant speaker separation. Furthermore, TF-CrossNet exhibits faster and more stable training in comparison to recent baselines. Additionally, TF-CrossNet's high performance extends to multi-microphone conditions, demonstrating its versatility in various acoustic scenarios.
- 1 November 2024
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 156 (5) , 3502-3513
Leveraging laryngograph data for robust voicing detection in speech
Accurately detecting voiced intervals in speech signals is a critical step in pitch tracking and has numerous applications. While conventional signal processing methods and deep learning algorithms have been proposed for this task, their need to fine-tune threshold parameters for different datasets and limited generalization restrict their utility in real-world applications. To address these challenges, this study proposes a supervised voicing detection model that leverages recorded laryngograph data. The model, adapted from a recently developed CrossNet architecture, is trained using reference voicing decisions derived from laryngograph datasets. Pretraining is also investigated to improve the generalization ability of the model. The proposed model produces robust voicing detection results, outperforming other strong baseline methods, and generalizes well to unseen datasets. The source code of the proposed model with pretraining is provided along with the list of used laryngograph datasets to facilitate further research in this area.
- 1 November 2024
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 156 (5) , 3502-3513
Leveraging laryngograph data for robust voicing detection in speech
Accurately detecting voiced intervals in speech signals is a critical step in pitch tracking and has numerous applications. While conventional signal processing methods and deep learning algorithms have been proposed for this task, their need to fine-tune threshold parameters for different datasets and limited generalization restrict their utility in real-world applications. To address these challenges, this study proposes a supervised voicing detection model that leverages recorded laryngograph data. The model, adapted from a recently developed CrossNet architecture, is trained using reference voicing decisions derived from laryngograph datasets. Pretraining is also investigated to improve the generalization ability of the model. The proposed model produces robust voicing detection results, outperforming other strong baseline methods, and generalizes well to unseen datasets. The source code of the proposed model with pretraining is provided along with the list of used laryngograph datasets to facilitate further research in this area.
- 25 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 2467-2476
Multi-Channel Conversational Speaker Separation via Neural Diarization
When dealing with overlapped speech, the performance of automatic speech recognition (ASR) systems substantially degrades as they are designed for single-talker speech. To enhance ASR performance in conversational or meeting environments, continuous speaker separation (CSS) is commonly employed. However, CSS requires a short separation window to avoid many speakers inside the window and sequential grouping of discontinuous speech segments. To address these limitations, we introduce a new multi-channel framework called “speaker separation via neural diarization” (SSND) for meeting environments. Our approach utilizes an end-to-end diarization system to identify the speech activity of each individual speaker. By leveraging estimated speaker boundaries, we generate a sequence of embeddings, which in turn facilitate the assignment of speakers to the outputs of a multi-talker separation model. SSND addresses the permutation ambiguity issue of talker-independent speaker separation during the diarization phase through location-based training, rather than during the separation process. This unique approach allows multiple non-overlapped speakers to be assigned to the same output stream, making it possible to efficiently process long segments—a task impossible with CSS. Additionally, SSND is naturally suitable for speaker-attributed ASR. We evaluate our proposed diarization and separation methods on the open LibriCSS dataset, advancing state-of-the-art diarization and ASR results by a large margin.
- 25 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 2467-2476
Multi-Channel Conversational Speaker Separation via Neural Diarization
When dealing with overlapped speech, the performance of automatic speech recognition (ASR) systems substantially degrades as they are designed for single-talker speech. To enhance ASR performance in conversational or meeting environments, continuous speaker separation (CSS) is commonly employed. However, CSS requires a short separation window to avoid many speakers inside the window and sequential grouping of discontinuous speech segments. To address these limitations, we introduce a new multi-channel framework called “speaker separation via neural diarization” (SSND) for meeting environments. Our approach utilizes an end-to-end diarization system to identify the speech activity of each individual speaker. By leveraging estimated speaker boundaries, we generate a sequence of embeddings, which in turn facilitate the assignment of speakers to the outputs of a multi-talker separation model. SSND addresses the permutation ambiguity issue of talker-independent speaker separation during the diarization phase through location-based training, rather than during the separation process. This unique approach allows multiple non-overlapped speakers to be assigned to the same output stream, making it possible to efficiently process long segments—a task impossible with CSS. Additionally, SSND is naturally suitable for speaker-attributed ASR. We evaluate our proposed diarization and separation methods on the open LibriCSS dataset, advancing state-of-the-art diarization and ASR results by a large margin.
- Open Access
Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR
It has been shown that the intelligibility of noisy speech can be improved by speech enhancement (SE) algorithms. However, monaural SE has not been established as an effective frontend for automatic speech recognition (ASR) in noisy conditions compared to an ASR model trained on noisy speech directly. The divide between SE and ASR impedes the progress of robust ASR systems, especially as SE has made major advances in recent years. This paper focuses on eliminating this divide with an ARN (attentive recurrent network) time-domain and a CrossNet time-frequency domain enhancement models. The proposed systems fully decouple frontend enhancement and backend ASR trained only on clean speech. Results on the WSJ, CHiME-2, LibriSpeech, and CHiME-4 corpora demonstrate that ARN and CrossNet enhanced speech both translate to improved ASR results in noisy and reverberant environments, and generalize well to real acoustic scenarios. The proposed system outperforms the baselines trained on corrupted speech directly. Furthermore, it cuts the previous best word error rate (WER) on CHiME-2 by $28.4\%$ relatively with a $5.57\%$ WER, and achieves $3.32/4.44\%$ WER on single-channel CHiME-4 simulated/real test data without training on CHiME-4.Open AccessTowards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR
It has been shown that the intelligibility of noisy speech can be improved by speech enhancement (SE) algorithms. However, monaural SE has not been established as an effective frontend for automatic speech recognition (ASR) in noisy conditions compared to an ASR model trained on noisy speech directly. The divide between SE and ASR impedes the progress of robust ASR systems, especially as SE has made major advances in recent years. This paper focuses on eliminating this divide with an ARN (attentive recurrent network) time-domain and a CrossNet time-frequency domain enhancement models. The proposed systems fully decouple frontend enhancement and backend ASR trained only on clean speech. Results on the WSJ, CHiME-2, LibriSpeech, and CHiME-4 corpora demonstrate that ARN and CrossNet enhanced speech both translate to improved ASR results in noisy and reverberant environments, and generalize well to real acoustic scenarios. The proposed system outperforms the baselines trained on corrupted speech directly. Furthermore, it cuts the previous best word error rate (WER) on CHiME-2 by $28.4\%$ relatively with a $5.57\%$ WER, and achieves $3.32/4.44\%$ WER on single-channel CHiME-4 simulated/real test data without training on CHiME-4. - Open Access
CrossNet: Leveraging Global, Cross-Band, Narrow-Band, and Positional Encoding for Single- and Multi-Channel Speaker Separation
We introduce CrossNet, a complex spectral mapping approach to speaker separation and enhancement in reverberant and noisy conditions. The proposed architecture comprises an encoder layer, a global multi-head self-attention module, a cross-band module, a narrow-band module, and an output layer. CrossNet captures global, cross-band, and narrow-band correlations in the time-frequency domain. To address performance degradation in long utterances, we introduce a random chunk positional encoding. Experimental results on multiple datasets demonstrate the effectiveness and robustness of CrossNet, achieving state-of-the-art performance in tasks including reverberant and noisy-reverberant speaker separation. Furthermore, CrossNet exhibits faster and more stable training in comparison to recent baselines. Additionally, CrossNet's high performance extends to multi-microphone conditions, demonstrating its versatility in various acoustic scenarios.
Open AccessCrossNet: Leveraging Global, Cross-Band, Narrow-Band, and Positional Encoding for Single- and Multi-Channel Speaker Separation
We introduce CrossNet, a complex spectral mapping approach to speaker separation and enhancement in reverberant and noisy conditions. The proposed architecture comprises an encoder layer, a global multi-head self-attention module, a cross-band module, a narrow-band module, and an output layer. CrossNet captures global, cross-band, and narrow-band correlations in the time-frequency domain. To address performance degradation in long utterances, we introduce a random chunk positional encoding. Experimental results on multiple datasets demonstrate the effectiveness and robustness of CrossNet, achieving state-of-the-art performance in tasks including reverberant and noisy-reverberant speaker separation. Furthermore, CrossNet exhibits faster and more stable training in comparison to recent baselines. Additionally, CrossNet's high performance extends to multi-microphone conditions, demonstrating its versatility in various acoustic scenarios.
- 28 February 2024
- Published by Elsevier in Neural Networks
- Vol. 173, 106209
Expansion of the editorial team
- 28 February 2024
- Published by Elsevier in Neural Networks
- Vol. 173, 106209
Expansion of the editorial team
- Open Access
Combined Generative and Predictive Modeling for Speech Super-resolution
Speech super-resolution (SR) is the task that restores high-resolution speech from low-resolution input. Existing models employ simulated data and constrained experimental settings, which limit generalization to real-world SR. Predictive models are known to perform well in fixed experimental settings, but can introduce artifacts in adverse conditions. On the other hand, generative models learn the distribution of target data and have a better capacity to perform well on unseen conditions. In this study, we propose a novel two-stage approach that combines the strengths of predictive and generative models. Specifically, we employ a diffusion-based model that is conditioned on the output of a predictive model. Our experiments demonstrate that the model significantly outperforms single-stage counterparts and existing strong baselines on benchmark SR datasets. Furthermore, we introduce a repainting technique during the inference of the diffusion process, enabling the proposed model to regenerate high-frequency components even in mismatched conditions. An additional contribution is the collection of and evaluation on real SR recordings, using the same microphone at different native sampling rates. We make this dataset freely accessible, to accelerate progress towards real-world speech super-resolution.
Open AccessCombined Generative and Predictive Modeling for Speech Super-resolution
Speech super-resolution (SR) is the task that restores high-resolution speech from low-resolution input. Existing models employ simulated data and constrained experimental settings, which limit generalization to real-world SR. Predictive models are known to perform well in fixed experimental settings, but can introduce artifacts in adverse conditions. On the other hand, generative models learn the distribution of target data and have a better capacity to perform well on unseen conditions. In this study, we propose a novel two-stage approach that combines the strengths of predictive and generative models. Specifically, we employ a diffusion-based model that is conditioned on the output of a predictive model. Our experiments demonstrate that the model significantly outperforms single-stage counterparts and existing strong baselines on benchmark SR datasets. Furthermore, we introduce a repainting technique during the inference of the diffusion process, enabling the proposed model to regenerate high-frequency components even in mismatched conditions. An additional contribution is the collection of and evaluation on real SR recordings, using the same microphone at different native sampling rates. We make this dataset freely accessible, to accelerate progress towards real-world speech super-resolution.