Shuai Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Shuai Wang has a total of 107 co-authors. Here are the top co-authors by number of shared publications:
- Haizhou Li
The Chinese University of Hong Kong, Shenzhen, China
17 shared publications - Yanmin Qian
Shanghai Jiao Tong University
7 shared publications - Kai Yu
Shanghai Jiao Tong University, Shanghai, China
5 shared publications Haizhou Li
School of Data Science; Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China
4 shared publications- Helen Meng
Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China
3 shared publications Ke Zhang
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
3 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 2461-2469
BeatEdit: Symbolic Music Generation as Explicit Editing
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 2461-2469
Open AccessBeatEdit: Symbolic Music Generation as Explicit Editing
- 12 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. PP, 1-14
MeMo: Attentional Momentum for Real-time Audio-visual Target Speaker Extraction under Impaired Visual Conditions
Audio-visual target speaker extraction (AV-TSE) faces significant performance degradation when visual cues are missing or impaired, limiting its practical deployment in real-world scenarios. This paper introduces MeMo, a novel framework that achieves attentional momentum through adaptive memory banks to maintain consistent focus on target speakers despite visual impairments. MeMo employs two complementary memory banks: a speaker bank storing identity features and a contextual bank capturing temporal speech patterns. The framework leverages self-enrolled speech from previous processing windows as supplementary reference, enabling robust extraction even when current visual cues are unreliable. Extensive experiments on the VoxCeleb2 dataset demonstrate MeMo's effectiveness, yielding approximately 27% relative improvement over the baseline model under impaired visual conditions in online streaming scenarios. The contextual bank proves particularly effective, providing over 2 dB gains across multiple evaluation metrics while maintaining real-time processing capabilities.
- 12 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. PP, 1-14
MeMo: Attentional Momentum for Real-time Audio-visual Target Speaker Extraction under Impaired Visual Conditions
Audio-visual target speaker extraction (AV-TSE) faces significant performance degradation when visual cues are missing or impaired, limiting its practical deployment in real-world scenarios. This paper introduces MeMo, a novel framework that achieves attentional momentum through adaptive memory banks to maintain consistent focus on target speakers despite visual impairments. MeMo employs two complementary memory banks: a speaker bank storing identity features and a contextual bank capturing temporal speech patterns. The framework leverages self-enrolled speech from previous processing windows as supplementary reference, enabling robust extraction even when current visual cues are unreliable. Extensive experiments on the VoxCeleb2 dataset demonstrate MeMo's effectiveness, yielding approximately 27% relative improvement over the baseline model under impaired visual conditions in online streaming scenarios. The contextual bank proves particularly effective, providing over 2 dB gains across multiple evaluation metrics while maintaining real-time processing capabilities.
- 12 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. PP, 1-12
Context-Aware Two-Stage Training for Domain Generalization in Speech Separation
Speech separation aims to isolate individual speech signals from multi-talker speech mixtures. Despite the impressive results on synthetic benchmarks, most systems suffer noticeable performance drop in face of real-world speech mixtures due to severe domain mismatch. To address this, we introduce a novel context-aware two-stage training scheme with iterative updates for speech separation models. In this training scheme, the conventional end-to-end architecture is replaced with a framework that contains a context extractor and a segregator. The two modules are trained iteratively to emulate the speech-separation process of the human auditory system. We evaluate the proposed training scheme through cross-domain experiments on both synthetic and real-world speech mixtures, and demonstrate that the proposed scheme effectively boosts separation quality across various domains without adaptation, as measured by signal quality metrics and word error rate (WER). Additionally, our ablation study on the real test set highlights that the perceptual objects, including phoneme and word representations from pretrained self-supervised learning (SSL) models, serve as effective training targets for separation models.
- 12 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. PP, 1-12
Context-Aware Two-Stage Training for Domain Generalization in Speech Separation
Speech separation aims to isolate individual speech signals from multi-talker speech mixtures. Despite the impressive results on synthetic benchmarks, most systems suffer noticeable performance drop in face of real-world speech mixtures due to severe domain mismatch. To address this, we introduce a novel context-aware two-stage training scheme with iterative updates for speech separation models. In this training scheme, the conventional end-to-end architecture is replaced with a framework that contains a context extractor and a segregator. The two modules are trained iteratively to emulate the speech-separation process of the human auditory system. We evaluate the proposed training scheme through cross-domain experiments on both synthetic and real-world speech mixtures, and demonstrate that the proposed scheme effectively boosts separation quality across various domains without adaptation, as measured by signal quality metrics and word error rate (WER). Additionally, our ablation study on the real test set highlights that the perceptual objects, including phoneme and word representations from pretrained self-supervised learning (SSL) models, serve as effective training targets for separation models.
- 29 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2801-2814
ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction
Audio-visual target speaker extraction (AV-TSE) models primarily rely on visual cues from the target speaker. However, humans also leverage linguistic knowledge, such as syntactic constraints, next word prediction, and prior knowledge of conversation, to extract target speech. Inspired by this observation, we propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models through three distinct guidance strategies: output linguistic constraints, intermediate linguistic prediction, and input linguistic prior. Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones demonstrate the effectiveness of our approach. Significant improvements are observed in challenging scenarios, including visual cue impaired, unseen languages, target speaker switches, increased interfering speakers, and out-of-domain test set.
- 29 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2801-2814
ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction
Audio-visual target speaker extraction (AV-TSE) models primarily rely on visual cues from the target speaker. However, humans also leverage linguistic knowledge, such as syntactic constraints, next word prediction, and prior knowledge of conversation, to extract target speech. Inspired by this observation, we propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models through three distinct guidance strategies: output linguistic constraints, intermediate linguistic prediction, and input linguistic prior. Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones demonstrate the effectiveness of our approach. Significant improvements are observed in challenging scenarios, including visual cue impaired, unseen languages, target speaker switches, increased interfering speakers, and out-of-domain test set.
- 23 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2763-2773
Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
The growing reliance on large-scale speech data has made privacy protection a critical concern. However, existing anonymization approaches often degrade data utility, for example by disrupting acoustic continuity or reducing vocal diversity, which compromises the value of speech data for downstream tasks such as Automatic Speech Recognition (ASR), Text-to- Speech (TTS), and Speech Emotion Recognition (SER). Current evaluation practices are also limited, as they mainly rely on direct testing of anonymized speech with pretrained models, providing only a partial view of utility. To address these issues, we propose a novel two-stage framework that protects both linguistic content and acoustic identity while maintaining usability. For content privacy, we employ a generative speech editing model to seamlessly replace personally identifiable information (PII), and for voice privacy, we introduce F3-VA, a flow-matchingbased anonymization framework with a three-stage design that produces diverse and distinct anonymized speakers. To enable a more comprehensive assessment, we evaluate privacy using both acoustic- and content-based speaker verification metrics, and assess utility by training ASR, TTS, and SER models from scratch. Experimental results show that our framework achieves stronger privacy protection with minimal utility degradation compared to baselines from the VoicePrivacy Challenge, while the proposed evaluation protocol provides a more realistic reflection of the utility of anonymized speech under privacy protection.
- 23 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2763-2773
Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
The growing reliance on large-scale speech data has made privacy protection a critical concern. However, existing anonymization approaches often degrade data utility, for example by disrupting acoustic continuity or reducing vocal diversity, which compromises the value of speech data for downstream tasks such as Automatic Speech Recognition (ASR), Text-to- Speech (TTS), and Speech Emotion Recognition (SER). Current evaluation practices are also limited, as they mainly rely on direct testing of anonymized speech with pretrained models, providing only a partial view of utility. To address these issues, we propose a novel two-stage framework that protects both linguistic content and acoustic identity while maintaining usability. For content privacy, we employ a generative speech editing model to seamlessly replace personally identifiable information (PII), and for voice privacy, we introduce F3-VA, a flow-matchingbased anonymization framework with a three-stage design that produces diverse and distinct anonymized speakers. To enable a more comprehensive assessment, we evaluate privacy using both acoustic- and content-based speaker verification metrics, and assess utility by training ASR, TTS, and SER models from scratch. Experimental results show that our framework achieves stronger privacy protection with minimal utility degradation compared to baselines from the VoicePrivacy Challenge, while the proposed evaluation protocol provides a more realistic reflection of the utility of anonymized speech under privacy protection.
- 10 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 246-258
PhiNet: Speaker Verification With Phonetic Interpretability
Despite remarkable progress, automatic speaker verification (ASV) systems typically lack the transparency required for high-accountability applications. Motivated by how human experts perform forensic speaker comparison (FSC), we propose a speaker verification network with phonetic interpretability, PhiNet, designed to enhance both local and global interpretability by leveraging phonetic evidence in decision-making. For users, PhiNet provides detailed phonetic-level comparisons that enable manual inspection of speaker-specific features and facilitate a more critical evaluation of verification outcomes. For developers, it offers explicit reasoning behind verification decisions, simplifying error tracing and informing hyperparameter selection. In our experiments, we demonstrate PhiNet’s interpretability with practical examples, including its application in analyzing the impact of different hyperparameters. We conduct both qualitative and quantitative evaluations of the proposed interpretability methods and assess speaker verification performance across multiple benchmark datasets, including VoxCeleb, SITW, and LibriSpeech. Results show that PhiNet achieves performance comparable to traditional black-box ASV models while offering meaningful, interpretable explanations for its decisions, bridging the gap between ASV and forensic analysis.
- 10 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 246-258
PhiNet: Speaker Verification With Phonetic Interpretability
Despite remarkable progress, automatic speaker verification (ASV) systems typically lack the transparency required for high-accountability applications. Motivated by how human experts perform forensic speaker comparison (FSC), we propose a speaker verification network with phonetic interpretability, PhiNet, designed to enhance both local and global interpretability by leveraging phonetic evidence in decision-making. For users, PhiNet provides detailed phonetic-level comparisons that enable manual inspection of speaker-specific features and facilitate a more critical evaluation of verification outcomes. For developers, it offers explicit reasoning behind verification decisions, simplifying error tracing and informing hyperparameter selection. In our experiments, we demonstrate PhiNet’s interpretability with practical examples, including its application in analyzing the impact of different hyperparameters. We conduct both qualitative and quantitative evaluations of the proposed interpretability methods and assess speaker verification performance across multiple benchmark datasets, including VoxCeleb, SITW, and LibriSpeech. Results show that PhiNet achieves performance comparable to traditional black-box ASV models while offering meaningful, interpretable explanations for its decisions, bridging the gap between ASV and forensic analysis.
- 5 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 311-323
Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation
Speech separation (SS) seeks to disentangle a multi-speaker speech mixture into single-speaker speech streams. Although SS can be generally achieved using offline methods, such a processing paradigm is not suitable for real-time streaming applications. Causal separation models, which rely only on past and present information,offer a promising solution for real-time streaming. However, these models typically suffer from notable performance degradation due to the absence of future context. In this paper, we introduce a novel frontend that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models during training through predictive patterns. The pretrained frontend employs a transformer decoder network with a causal convolutional encoder as the backbone and is pretrained in a self-supervised manner with two innovative pretext tasks: autoregressive hybrid prediction and contextual knowledge distillation. These tasks allow the model to capture predictive patterns directly from mixtures in a self-supervised manner. The pretrained frontend subsequently serves as a feature extractor to generate high-quality predictive patterns. Comprehensive evaluations on synthetic and real-world datasets are carried out to validate the effectiveness of the proposed pretrained frontend.
- 5 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 311-323
Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation
Speech separation (SS) seeks to disentangle a multi-speaker speech mixture into single-speaker speech streams. Although SS can be generally achieved using offline methods, such a processing paradigm is not suitable for real-time streaming applications. Causal separation models, which rely only on past and present information,offer a promising solution for real-time streaming. However, these models typically suffer from notable performance degradation due to the absence of future context. In this paper, we introduce a novel frontend that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models during training through predictive patterns. The pretrained frontend employs a transformer decoder network with a causal convolutional encoder as the backbone and is pretrained in a self-supervised manner with two innovative pretext tasks: autoregressive hybrid prediction and contextual knowledge distillation. These tasks allow the model to capture predictive patterns directly from mixtures in a self-supervised manner. The pretrained frontend subsequently serves as a feature extractor to generate high-quality predictive patterns. Comprehensive evaluations on synthetic and real-world datasets are carried out to validate the effectiveness of the proposed pretrained frontend.
- 4 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 33 (10709908) , 351-355
TPEech: Target Speaker Extraction and Noise Suppression With Historical Dialogue Text Cues
In complex multi-speaker scenarios with significant speaker overlap and background noise, extracting the target speaker's speech remains a major challenge. This capability is crucial for dialogue-based applications such as AI speech assistants, where downstream tasks such as speech recognition depend on clean speech. A potential solution to address these challenges is Target Speaker Extraction (TSE), which leverages auxiliary information to extract target speech from mixed and noisy speech, thus overcoming the limitations of Speech Separation (SS) and Speech Enhancement (SE). In particular, we propose a multi-modal TSE network, namely Text Prompt Extractor with echo cue block (TPEech), which uses historical dialogue text as cues for extraction and incorporates the echo cue block (ECB) to further exploit this cue and enhance TSE performance. The experiments show the excellent extraction and denoising capabilities of our proposed network. TPEech achieves an SI-SDRi of 9.632 dB, an SDR of 13.045 dB, a PESQ of 2.814, and a STOI of 0.885, outperforming competitive baselines. Additionally, we experimentally verify that TPEech is robust against semantically incomplete textual prompts. Dataset and source code will be publicly available.
- 4 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 33 (10709908) , 351-355
TPEech: Target Speaker Extraction and Noise Suppression With Historical Dialogue Text Cues
In complex multi-speaker scenarios with significant speaker overlap and background noise, extracting the target speaker's speech remains a major challenge. This capability is crucial for dialogue-based applications such as AI speech assistants, where downstream tasks such as speech recognition depend on clean speech. A potential solution to address these challenges is Target Speaker Extraction (TSE), which leverages auxiliary information to extract target speech from mixed and noisy speech, thus overcoming the limitations of Speech Separation (SS) and Speech Enhancement (SE). In particular, we propose a multi-modal TSE network, namely Text Prompt Extractor with echo cue block (TPEech), which uses historical dialogue text as cues for extraction and incorporates the echo cue block (ECB) to further exploit this cue and enhance TSE performance. The experiments show the excellent extraction and denoising capabilities of our proposed network. TPEech achieves an SI-SDRi of 9.632 dB, an SDR of 13.045 dB, a PESQ of 2.814, and a STOI of 0.885, outperforming competitive baselines. Additionally, we experimentally verify that TPEech is robust against semantically incomplete textual prompts. Dataset and source code will be publicly available.