Heng Lu
Research Subjects
Research Interests
Heng Lu has a total of 63 co-authors. Here are the top co-authors by number of shared publications:
- Hongbin Zhou
Ximalaya Inc China
5 shared publications - Lei Xie
Speech and Language Processing Group (ASLP@NPU) Northwestern Polytechnical University Xi'an China
3 shared publications - Cunhang Fan
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, School of Computer Science and Technology, Anhui University, Hefei, Anhui, China
2 shared publications - Jianhao Ye
Ximalaya Inc China
2 shared publications - Jianhua Tao
Tsinghua University, Beijing, China
2 shared publications - Kaimeng Ren
Ximalaya Inc
2 shared publications
Latest Publications
- Top Cited Papers
- 27 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1243-1254
Vec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation
Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-quality texts and images in various tasks. While current speech LMs have made significant progress, there are still challenges to overcome in terms of achieving optimal speech quality and broad task generalization. This paper presents Vec-Tok Speech, an extensible framework that resembles multiple speech generation tasks, generating expressive and high-fidelity speech. Specifically, we propose a novel speech codec based on speech vectors and semantic tokens. Speech vectors contain acoustic details contributing to high-fidelity speech reconstruction, while semantic tokens focus on the linguistic content of speech, facilitating language modeling. Based on the proposed speech codec, Vec-Tok Speech leverages an LM to undertake the core of speech generation. Moreover, Byte-Pair Encoding (BPE) is introduced to reduce the token length and bit rate for lower exposure bias and longer context, improving the performance of LMs. Vec-Tok Speech can be used for intra- and cross-lingual zero-shot voice conversion (VC), zero-shot speaking style transfer text-to-speech (TTS), speech-to-speech translation (S2ST), speech denoising, and speaker de-identification and anonymization. Experiments show that Vec-Tok Speech, built on 50,000 hours of speech, performs better than other SOTA models.
- 27 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1243-1254
Top Cited PapersVec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation
Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-quality texts and images in various tasks. While current speech LMs have made significant progress, there are still challenges to overcome in terms of achieving optimal speech quality and broad task generalization. This paper presents Vec-Tok Speech, an extensible framework that resembles multiple speech generation tasks, generating expressive and high-fidelity speech. Specifically, we propose a novel speech codec based on speech vectors and semantic tokens. Speech vectors contain acoustic details contributing to high-fidelity speech reconstruction, while semantic tokens focus on the linguistic content of speech, facilitating language modeling. Based on the proposed speech codec, Vec-Tok Speech leverages an LM to undertake the core of speech generation. Moreover, Byte-Pair Encoding (BPE) is introduced to reduce the token length and bit rate for lower exposure bias and longer context, improving the performance of LMs. Vec-Tok Speech can be used for intra- and cross-lingual zero-shot voice conversion (VC), zero-shot speaking style transfer text-to-speech (TTS), speech-to-speech translation (S2ST), speech denoising, and speaker de-identification and anonymization. Experiments show that Vec-Tok Speech, built on 50,000 hours of speech, performs better than other SOTA models.
- 7 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 1506-1518
METTS: Multilingual Emotional Text-to-Speech by Cross-Speaker and Cross-Lingual Emotion Transfer
Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer – the heavy entanglement of speaker timbre , emotion and language factors in the speech signal will make a system to produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes a Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.
- 7 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 1506-1518
METTS: Multilingual Emotional Text-to-Speech by Cross-Speaker and Cross-Lingual Emotion Transfer
Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer – the heavy entanglement of speaker timbre , emotion and language factors in the speech signal will make a system to produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes a Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 19-26
Exploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 19-26
Open AccessExploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 51-57
Multimodal Cross-Lingual Features and Weight Fusion for Cross-Cultural Humor Detection
Sentiment analysis plays a crucial role in interpreting human interactions across different modalities. This paper addresses the Cross-Cultural Multilingual Humour Detection Sub-Challenge in the MuSe 2023 Multi-modal Sentiment Analysis Challenge, which aims to identify humor instances in cross-cultural situations using multimodal data. In this paper, we explore the unique window length processing of audio data to more accurately and fully capture humor information. Furthermore, we employ a data augmentation technique for text to bridge the gap between the train set and the test set. This augmentation involves converting German and English samples in the database to each other, as well as translating both languages into Chinese. These techniques aim to enhance the model's generalization across different languages and alleviate the scarcity of humor samples. For multimodal fusion, we use the late fusion technique applied to different Gated Recurrent Unit (GRU) models with modality-specific weights obtained through gradient descent. Experimental results demonstrate the effectiveness of our approach, achieving an Area Under the Curve (AUC) score of 0.872 on the MuSe-Humor test set. This performance ranked second place in the sub-challenge.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 51-57
Open AccessMultimodal Cross-Lingual Features and Weight Fusion for Cross-Cultural Humor Detection
Sentiment analysis plays a crucial role in interpreting human interactions across different modalities. This paper addresses the Cross-Cultural Multilingual Humour Detection Sub-Challenge in the MuSe 2023 Multi-modal Sentiment Analysis Challenge, which aims to identify humor instances in cross-cultural situations using multimodal data. In this paper, we explore the unique window length processing of audio data to more accurately and fully capture humor information. Furthermore, we employ a data augmentation technique for text to bridge the gap between the train set and the test set. This augmentation involves converting German and English samples in the database to each other, as well as translating both languages into Chinese. These techniques aim to enhance the model's generalization across different languages and alleviate the scarcity of humor samples. For multimodal fusion, we use the late fusion technique applied to different Gated Recurrent Unit (GRU) models with modality-specific weights obtained through gradient descent. Experimental results demonstrate the effectiveness of our approach, achieving an Area Under the Curve (AUC) score of 0.872 on the MuSe-Humor test set. This performance ranked second place in the sub-challenge.
- Open Access
- 30 July 2023
- Published by Darcy & Roy Press Co. Ltd. in Highlights in Science, Engineering and Technology
- Vol. 61, 188-191
Spatial-Temporal Preserving Multimodal Algorithm for EEG-based Emotion Recognition
Emotion recognition has been the focus of some human computer interaction researchers. Designing an efficient algorithm for emotion recognition is crucial for its potential use in HCI. Many algorithms have been developed for emotion recognition and a majority of them is based on electroencephalography (EEG). Despite the success of spatial–temporal recurrent neural network (STRNN) and multimodal deep neural networks, they can still be improved. This paper proposes an algorithm which tries to combine STRNN with multimodal deep neural network, where EEG signals are classified using STRNN and eye movement signals are classified using SVM. The results from the two classifiers are fused by max fusion. This paper shows that by taking spatial and temporal information of EEG signals into consideration and using multimodal fusion, the performance of emotion recognition can be improved.
- 30 July 2023
- Published by Darcy & Roy Press Co. Ltd. in Highlights in Science, Engineering and Technology
- Vol. 61, 188-191
Open AccessSpatial-Temporal Preserving Multimodal Algorithm for EEG-based Emotion Recognition
Emotion recognition has been the focus of some human computer interaction researchers. Designing an efficient algorithm for emotion recognition is crucial for its potential use in HCI. Many algorithms have been developed for emotion recognition and a majority of them is based on electroencephalography (EEG). Despite the success of spatial–temporal recurrent neural network (STRNN) and multimodal deep neural networks, they can still be improved. This paper proposes an algorithm which tries to combine STRNN with multimodal deep neural network, where EEG signals are classified using STRNN and eye movement signals are classified using SVM. The results from the two classifiers are fused by max fusion. This paper shows that by taking spatial and temporal information of EEG signals into consideration and using multimodal fusion, the performance of emotion recognition can be improved.
- Open Access
METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer
Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes the Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift-based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization-based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.
Open AccessMETTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer
Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes the Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift-based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization-based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.
- 10 May 2023
- Published by Springer Nature
- p. 306-317
Automatic Stress Annotation and Prediction for Expressive Mandarin TTS
- 10 May 2023
- Published by Springer Nature
- p. 306-317
Automatic Stress Annotation and Prediction for Expressive Mandarin TTS
- 10 May 2023
- Published by Springer Nature
- p. 196-207
Improving Fine-Grained Emotion Control and Transfer with Gated Emotion Representations in Speech Synthesis
- 10 May 2023
- Published by Springer Nature
- p. 196-207
Improving Fine-Grained Emotion Control and Transfer with Gated Emotion Representations in Speech Synthesis
- 16 June 2022
- Published by Elsevier in Gastroenterology
- Vol. 162 (7)
Tu1121: INDUCTION OF FGFR4 BY H. PYLORI MEDIATES ACTIVATION OF STAT3 THROUGH SRC SIGNALING IN GASTRIC CANCER
- 16 June 2022
- Published by Elsevier in Gastroenterology
- Vol. 162 (7)
Tu1121: INDUCTION OF FGFR4 BY H. PYLORI MEDIATES ACTIVATION OF STAT3 THROUGH SRC SIGNALING IN GASTRIC CANCER
- Open Access
FeatherTTS: Robust and Efficient attention based Neural TTS
Attention based neural TTS is elegant speech synthesis pipeline and has shown a powerful ability to generate natural speech. However, it is still not robust enough to meet the stability requirements for industrial products. Besides, it suffers from slow inference speed owning to the autoregressive generation process. In this work, we propose FeatherTTS, a robust and efficient attention-based neural TTS system. Firstly, we propose a novel Gaussian attention which utilizes interpretability of Gaussian attention and the strict monotonic property in TTS. By this method, we replace the commonly used stop token prediction architecture with attentive stop prediction. Secondly, we apply block sparsity on the autoregressive decoder to speed up speech synthesis. The experimental results show that our proposed FeatherTTS not only nearly eliminates the problem of word skipping, repeating in particularly hard texts and keep the naturalness of generated speech, but also speeds up acoustic feature generation by 3.5 times over Tacotron. Overall, the proposed FeatherTTS can be $35$x faster than real-time on a single CPU.
Open AccessFeatherTTS: Robust and Efficient attention based Neural TTS
Attention based neural TTS is elegant speech synthesis pipeline and has shown a powerful ability to generate natural speech. However, it is still not robust enough to meet the stability requirements for industrial products. Besides, it suffers from slow inference speed owning to the autoregressive generation process. In this work, we propose FeatherTTS, a robust and efficient attention-based neural TTS system. Firstly, we propose a novel Gaussian attention which utilizes interpretability of Gaussian attention and the strict monotonic property in TTS. By this method, we replace the commonly used stop token prediction architecture with attentive stop prediction. Secondly, we apply block sparsity on the autoregressive decoder to speed up speech synthesis. The experimental results show that our proposed FeatherTTS not only nearly eliminates the problem of word skipping, repeating in particularly hard texts and keep the naturalness of generated speech, but also speeds up acoustic feature generation by 3.5 times over Tacotron. Overall, the proposed FeatherTTS can be $35$x faster than real-time on a single CPU.