Ruibo Fu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Ruibo Fu has a total of 142 co-authors. Here are the top co-authors by number of shared publications:
- Jianhua Tao
Tsinghua University, Beijing, China
21 shared publications Zhengqi Wen
Tsinghua University
18 shared publications- Jiangyan Yi
Department of Automation, Tsinghua University, Beijing, China
13 shared publications Xuefei Liu
Chinese Academy of Sciences
9 shared publicationsJianhua Tao
8 shared publicationsYukun Liu
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
8 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 17730-17734
AT-ADD: All-Type Audio Deepfake Detection Challenge Summary
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 17730-17734
Open AccessAT-ADD: All-Type Audio Deepfake Detection Challenge Summary
- 30 September 2026
- Published by Elsevier in Artificial Intelligence
- Vol. 361
P2Mark: Plug-and-play parameter-level watermarking for neural speech generation
- 30 September 2026
- Published by Elsevier in Artificial Intelligence
- Vol. 361
P2Mark: Plug-and-play parameter-level watermarking for neural speech generation
- Top Cited Papers
- 25 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2077-2089
SSE-Net: Toward Low-Power-Consumption Spiking Neural Network for Monaural Speech Enhancement
Speech enhancement (SE) as a front-end task, especially in edge device deployment, requires low complexity and power consumption. However, deploying most artificial neural network (ANN)-based SE models while ensuring performance is challenging, particularly with the rapid development of large models in recent years. Spiking neural networks (SNNs) have shown potential in reducing power consumption. However, SNN-based SE models face two main challenges: on one hand, directly converting an ANN model to a SNN model introduces redundant modules, leading to difficulties in model training and information mismatch; on the other hand, the discrete binary activation and complex spatio-temporal dynamics of SNNs often result in information loss. To address these challenges, this paper proposes the SNN Speech Enhancement Network (SSE-Net). Unlike conventional approaches that convert ANNs to SNNs, all modules in SSE-Net are specifically designed for spike signal characteristics. Furthermore, the method innovatively develops a spiking feature extraction group that simultaneously converts information into spike signals while extracting original speech features, thereby effectively reducing information loss while significantly lowering both power consumption and computational complexity. The proposed method further designs information transformation blocks to convert spike signals back into continuous signals, fine-tuning and supplementing the speech information. Experimental results demonstrate that our proposed model achieves state of the art (SOTA) performance compared to the best SNN-SE models, with a 62% reduction in power proxy and 11% reduction in power cost, it also demonstrates strong competitiveness when compared with ANN-based SE models.
- 25 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2077-2089
Top Cited PapersSSE-Net: Toward Low-Power-Consumption Spiking Neural Network for Monaural Speech Enhancement
Speech enhancement (SE) as a front-end task, especially in edge device deployment, requires low complexity and power consumption. However, deploying most artificial neural network (ANN)-based SE models while ensuring performance is challenging, particularly with the rapid development of large models in recent years. Spiking neural networks (SNNs) have shown potential in reducing power consumption. However, SNN-based SE models face two main challenges: on one hand, directly converting an ANN model to a SNN model introduces redundant modules, leading to difficulties in model training and information mismatch; on the other hand, the discrete binary activation and complex spatio-temporal dynamics of SNNs often result in information loss. To address these challenges, this paper proposes the SNN Speech Enhancement Network (SSE-Net). Unlike conventional approaches that convert ANNs to SNNs, all modules in SSE-Net are specifically designed for spike signal characteristics. Furthermore, the method innovatively develops a spiking feature extraction group that simultaneously converts information into spike signals while extracting original speech features, thereby effectively reducing information loss while significantly lowering both power consumption and computational complexity. The proposed method further designs information transformation blocks to convert spike signals back into continuous signals, fine-tuning and supplementing the speech information. Experimental results demonstrate that our proposed model achieves state of the art (SOTA) performance compared to the best SNN-SE models, with a 62% reduction in power proxy and 11% reduction in power cost, it also demonstrates strong competitiveness when compared with ANN-based SE models.
- 11 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 1300-1311
DegDiT: Controllable Audio Generation With Dynamic Event Graph Guided Diffusion Transformer
Controllable text-to-audio generation aims to synthesize audio from textual descriptions while satisfying user-specified constraints, including event types, onset and offset timestamps, and temporal sequences. This enables precise control over both the content and temporal structure of the generated audio. Despite recent progress, existing methods still face inherent trade-offs among accurate temporal localization, open-vocabulary scalability, and practical efficiency. To address these challenges, we propose DegDiT, a novel dynamic event graph-guided diffusion transformer framework for open-vocabulary controllable audio generation. DegDiT encodes the events from the text description into structured dynamic graphs. The nodes represent distinct audio events, while edges encode the temporal relationships between them. A transformer is employed to integrate these graphs and produce contextualized event embeddings that serve as guidance for the diffusion model. To ensure high-quality and diverse training data, we introduce a quality-balanced data selection pipeline that combines hierarchical event annotation with multi-criteria quality scoring, resulting in a curated dataset with semantic diversity. Furthermore, we present consensus preference optimization, facilitating audio generation through consensus among multiple reward signals. Extensive experiments on AudioCondition, DESED, and AudioTime datasets demonstrate that DegDiT achieves state-of-the-art performances across a variety of objective and subjective evaluation metrics.
- 11 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 1300-1311
DegDiT: Controllable Audio Generation With Dynamic Event Graph Guided Diffusion Transformer
Controllable text-to-audio generation aims to synthesize audio from textual descriptions while satisfying user-specified constraints, including event types, onset and offset timestamps, and temporal sequences. This enables precise control over both the content and temporal structure of the generated audio. Despite recent progress, existing methods still face inherent trade-offs among accurate temporal localization, open-vocabulary scalability, and practical efficiency. To address these challenges, we propose DegDiT, a novel dynamic event graph-guided diffusion transformer framework for open-vocabulary controllable audio generation. DegDiT encodes the events from the text description into structured dynamic graphs. The nodes represent distinct audio events, while edges encode the temporal relationships between them. A transformer is employed to integrate these graphs and produce contextualized event embeddings that serve as guidance for the diffusion model. To ensure high-quality and diverse training data, we introduce a quality-balanced data selection pipeline that combines hierarchical event annotation with multi-criteria quality scoring, resulting in a curated dataset with semantic diversity. Furthermore, we present consensus preference optimization, facilitating audio generation through consensus among multiple reward signals. Extensive experiments on AudioCondition, DESED, and AudioTime datasets demonstrate that DegDiT achieves state-of-the-art performances across a variety of objective and subjective evaluation metrics.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28, 15490-15501
Multimodal Diffusion Transformer With Memory Bank for Scalable Long-Duration Talking Video Generation
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28, 15490-15501
Multimodal Diffusion Transformer With Memory Bank for Scalable Long-Duration Talking Video Generation
- 18 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 339-351
RPRA-ADD: Forgery Trace Enhancement-Driven Audio Deepfake Detection
Existing methods for deepfake audio detection have exhibited limited efficacy. However, they still face challenges in generalizing to new forgery techniques and evolving attack patterns. This limitation mainly stems from the fact that the models rely heavily on the distribution of the training data and struggle to learn a decision boundary that discerns the essential characteristics of forgeries. Additionally, relying solely on a classification loss impedes the ability to extract the intrinsic discrepancies between real and fake audio. In this paper, we propose the RPRA-ADD, an integrated Reconstruction-Perception-Reinforcement-Attention networks based forgery trace enhancement-driven robust audio deepfake detection framework. First, we introduce a Global-Local Forgery Perception (GLFP) module for augmenting the acoustic perception capacity of forgery traces. To significantly accentuate the feature space distribution divergences between real and fake audio, the Multi-stage Dispersed Enhancement Loss (MDEL) is designed, which employs a dispersal strategy in multi-stage feature spaces. Furthermore, in order to sharpen feature awareness towards forgery traces, the Fake Trace Focused Attention (FTFA) mechanism is introduced to adjust attention weights dynamically guided by the reconstruction discrepancy matrix. Visualization experiments reveal that FTFA not only refines attention to voice segments, but also bolsters the generalization capability. Experimental results indicate that the proposed method achieves state-of-the-art performance on 4 benchmark datasets, including ASVspoof2019, ASVspoof2021, CodecFake, and FakeSound, yielding over 20% performance improvement. In addition, it surpasses existing methods in rigorous 3×3 cross-domain evaluations across Speech, Sound, and Singing, underscoring strong generalization capability across diverse audio domains.
- 18 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 339-351
RPRA-ADD: Forgery Trace Enhancement-Driven Audio Deepfake Detection
Existing methods for deepfake audio detection have exhibited limited efficacy. However, they still face challenges in generalizing to new forgery techniques and evolving attack patterns. This limitation mainly stems from the fact that the models rely heavily on the distribution of the training data and struggle to learn a decision boundary that discerns the essential characteristics of forgeries. Additionally, relying solely on a classification loss impedes the ability to extract the intrinsic discrepancies between real and fake audio. In this paper, we propose the RPRA-ADD, an integrated Reconstruction-Perception-Reinforcement-Attention networks based forgery trace enhancement-driven robust audio deepfake detection framework. First, we introduce a Global-Local Forgery Perception (GLFP) module for augmenting the acoustic perception capacity of forgery traces. To significantly accentuate the feature space distribution divergences between real and fake audio, the Multi-stage Dispersed Enhancement Loss (MDEL) is designed, which employs a dispersal strategy in multi-stage feature spaces. Furthermore, in order to sharpen feature awareness towards forgery traces, the Fake Trace Focused Attention (FTFA) mechanism is introduced to adjust attention weights dynamically guided by the reconstruction discrepancy matrix. Visualization experiments reveal that FTFA not only refines attention to voice segments, but also bolsters the generalization capability. Experimental results indicate that the proposed method achieves state-of-the-art performance on 4 benchmark datasets, including ASVspoof2019, ASVspoof2021, CodecFake, and FakeSound, yielding over 20% performance improvement. In addition, it surpasses existing methods in rigorous 3×3 cross-domain evaluations across Speech, Sound, and Singing, underscoring strong generalization capability across diverse audio domains.
- 14 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 4808-4820
A Joint Training Framework for Noise-Robust Speech Recognition Through Multi-Level Feature Fusion
In robust speech recognition systems, speech enhancement is commonly employed as a front-end preprocessing module to eliminate noise components from speech segments. However, this process inevitably introduces speech distortion while removing noise, leading to the loss of speech information, therefore causing performance decline in ASR. To address this issue, this paper proposes an innovative multi-level feature fusion (MLFF)architecture. First, during the speech enhancement stage, a multi-head self-attention and convolutional fusion network (MCF-Net) integrates the masking features and mapping features generated by enhancement processing, achieving preliminary alleviation of speech distortion. Subsequently, a cross-information fusion network (CIF-Net) performs secondary fusion between the MCF-Net output features and original noisy features to further mitigate distortion effects. Finally, last-stage fusion is conducted through an adaptive feature fusion network (AFF Net) between the abstract representations from the ASR encoder and the hidden layer features from intermediate speech enhancement stages, comprehensively compensating for information loss across processing stages. Experimental results on AISHELL-1 (Chinese) and LibriSpeech 100h (English) datasets demonstrate that our method achieves state-of-the-art (SOTA) performance in CER/WER metrics, validating the robustness and effective ness of the proposed architecture. Source code is available at https://github.com/fchest/MLFF.
- 14 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 4808-4820
A Joint Training Framework for Noise-Robust Speech Recognition Through Multi-Level Feature Fusion
In robust speech recognition systems, speech enhancement is commonly employed as a front-end preprocessing module to eliminate noise components from speech segments. However, this process inevitably introduces speech distortion while removing noise, leading to the loss of speech information, therefore causing performance decline in ASR. To address this issue, this paper proposes an innovative multi-level feature fusion (MLFF)architecture. First, during the speech enhancement stage, a multi-head self-attention and convolutional fusion network (MCF-Net) integrates the masking features and mapping features generated by enhancement processing, achieving preliminary alleviation of speech distortion. Subsequently, a cross-information fusion network (CIF-Net) performs secondary fusion between the MCF-Net output features and original noisy features to further mitigate distortion effects. Finally, last-stage fusion is conducted through an adaptive feature fusion network (AFF Net) between the abstract representations from the ASR encoder and the hidden layer features from intermediate speech enhancement stages, comprehensively compensating for information loss across processing stages. Experimental results on AISHELL-1 (Chinese) and LibriSpeech 100h (English) datasets demonstrate that our method achieves state-of-the-art (SOTA) performance in CER/WER metrics, validating the robustness and effective ness of the proposed architecture. Source code is available at https://github.com/fchest/MLFF.
- 31 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2596-2609
SpeechPalette: A Comprehensive Speech Editing Method for Text-Based Speech Editing, One-Shot TTS and Attributes Editing
Speech editing has garnered more and more attention due to its diverse applications. However, existing systems often require substantial manual effort or have limited capabilities in attribute editing, imposing significant constraints. In this work, we present SpeechPalette, a comprehensive high-quality speech editing method that allows users to easily modify various attributes of the selected speech segment according to their preferences. Specifically, the proposed model approaches speech editing from a decoupling perspective, disentangling critical information such as text, pitch, duration and more from the input speech. Then, reconstruction is achieved through a mask and prediction mechanism. Furthermore, we leverage a diffusion model to predict the residuals between the real and predicted speech, further enhancing synthesis quality. The proposed method not only excels at text-based speech editing but also handles tasks involving pitch and speed rate adjustments. Moreover, it also demonstrates remarkable performance in one-shot text-to-speech scenarios. While recent large-scale models achieve impressive synthesis quality through massive computational resources, SpeechPalette offers a balanced approach with explicit fine-grained control over speech attributes, practical deployment requirements, and competitive performance relative to similarly-sized systems. Experimental results across a range of tasks consistently demonstrate the superior performance of our method compared to baseline systems. Additionally, comprehensive ablation studies validate the effectiveness of our proposed approach.
- 31 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 2596-2609
SpeechPalette: A Comprehensive Speech Editing Method for Text-Based Speech Editing, One-Shot TTS and Attributes Editing
Speech editing has garnered more and more attention due to its diverse applications. However, existing systems often require substantial manual effort or have limited capabilities in attribute editing, imposing significant constraints. In this work, we present SpeechPalette, a comprehensive high-quality speech editing method that allows users to easily modify various attributes of the selected speech segment according to their preferences. Specifically, the proposed model approaches speech editing from a decoupling perspective, disentangling critical information such as text, pitch, duration and more from the input speech. Then, reconstruction is achieved through a mask and prediction mechanism. Furthermore, we leverage a diffusion model to predict the residuals between the real and predicted speech, further enhancing synthesis quality. The proposed method not only excels at text-based speech editing but also handles tasks involving pitch and speed rate adjustments. Moreover, it also demonstrates remarkable performance in one-shot text-to-speech scenarios. While recent large-scale models achieve impressive synthesis quality through massive computational resources, SpeechPalette offers a balanced approach with explicit fine-grained control over speech attributes, practical deployment requirements, and competitive performance relative to similarly-sized systems. Experimental results across a range of tasks consistently demonstrate the superior performance of our method compared to baseline systems. Additionally, comprehensive ablation studies validate the effectiveness of our proposed approach.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 12957-12964
MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 12957-12964
Open AccessMDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics
- Open Access
- 26 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 69-74
Less is More? Textual-Only Language Model for AVI challenge 2025
- 26 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 69-74
Open AccessLess is More? Textual-Only Language Model for AVI challenge 2025