HLHeng Lu

Heng Lu

*This is an algorithmically generated author record based on publicly available information.

Heng Lu has a total of 63 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers

    Vec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation

    by Xinfa Zhu, Yuanjun Lv, Yi Lei, Tao Li, Wendi He, Hongbin Zhou, Heng Lu, Lei Xie

    Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-quality texts and images in various tasks. While current speech LMs have made significant progress, there are still challenges to overcome in terms of achieving optimal speech quality and broad task generalization. This paper presents Vec-Tok Speech, an extensible framework that resembles multiple speech generation tasks, generating expressive and high-fidelity speech. Specifically, we propose a novel speech codec based on speech vectors and semantic tokens. Speech vectors contain acoustic details contributing to high-fidelity speech reconstruction, while semantic tokens focus on the linguistic content of speech, facilitating language modeling. Based on the proposed speech codec, Vec-Tok Speech leverages an LM to undertake the core of speech generation. Moreover, Byte-Pair Encoding (BPE) is introduced to reduce the token length and bit rate for lower exposure bias and longer context, improving the performance of LMs. Vec-Tok Speech can be used for intra- and cross-lingual zero-shot voice conversion (VC), zero-shot speaking style transfer text-to-speech (TTS), speech-to-speech translation (S2ST), speech denoising, and speaker de-identification and anonymization. Experiments show that Vec-Tok Speech, built on 50,000 hours of speech, performs better than other SOTA models.


  • METTS: Multilingual Emotional Text-to-Speech by Cross-Speaker and Cross-Lingual Emotion Transfer

    by Xinfa Zhu, Yi Lei, Tao Li, Yongmao Zhang, Hongbin Zhou, Heng Lu, Lei Xie

    Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer – the heavy entanglement of speaker timbre , emotion and language factors in the speech signal will make a system to produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes a Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.


  • Open Access

    Exploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction

    by Guofeng Yi, Yuguang Yang, Yu Pan, Yuhang Cao, Jixun Yao, Xiang Lv, Cunhang Fan, Zhao Lv, Jianhua Tao, Shan Liang, et al.


  • Open Access

    Multimodal Cross-Lingual Features and Weight Fusion for Cross-Cultural Humor Detection

    by Heng Xie, Jizhou Cui, Yuhang Cao, Junjie Chen, Jianhua Tao, Cunhang Fan, Xuefei Liu, Zhengqi Wen, Heng Lu, Yuguang Yang, et al.

    Sentiment analysis plays a crucial role in interpreting human interactions across different modalities. This paper addresses the Cross-Cultural Multilingual Humour Detection Sub-Challenge in the MuSe 2023 Multi-modal Sentiment Analysis Challenge, which aims to identify humor instances in cross-cultural situations using multimodal data. In this paper, we explore the unique window length processing of audio data to more accurately and fully capture humor information. Furthermore, we employ a data augmentation technique for text to bridge the gap between the train set and the test set. This augmentation involves converting German and English samples in the database to each other, as well as translating both languages into Chinese. These techniques aim to enhance the model's generalization across different languages and alleviate the scarcity of humor samples. For multimodal fusion, we use the late fusion technique applied to different Gated Recurrent Unit (GRU) models with modality-specific weights obtained through gradient descent. Experimental results demonstrate the effectiveness of our approach, achieving an Area Under the Curve (AUC) score of 0.872 on the MuSe-Humor test set. This performance ranked second place in the sub-challenge.


  • Open Access

    Spatial-Temporal Preserving Multimodal Algorithm for EEG-based Emotion Recognition

    by Heng Lu

    Emotion recognition has been the focus of some human computer interaction researchers. Designing an efficient algorithm for emotion recognition is crucial for its potential use in HCI. Many algorithms have been developed for emotion recognition and a majority of them is based on electroencephalography (EEG). Despite the success of spatial–temporal recurrent neural network (STRNN) and multimodal deep neural networks, they can still be improved. This paper proposes an algorithm which tries to combine STRNN with multimodal deep neural network, where EEG signals are classified using STRNN and eye movement signals are classified using SVM. The results from the two classifiers are fused by max fusion. This paper shows that by taking spatial and temporal information of EEG signals into consideration and using multimodal fusion, the performance of emotion recognition can be improved.

  • Open Access

    METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer

    by Xinfa Zhu, Yi Lei, Tao Li, Yongmao Zhang, Hongbin Zhou, Heng Lu, Lei Xie

    Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes the Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift-based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization-based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.

  • Automatic Stress Annotation and Prediction for Expressive Mandarin TTS

    by Wendi He, Yiting Lin, Jianhao Ye, Hongbin Zhou, Kaimeng Ren, Tianwei He, Pengfei Tan, Heng Lu

  • Tu1121: INDUCTION OF FGFR4 BY H. PYLORI MEDIATES ACTIVATION OF STAT3 THROUGH SRC SIGNALING IN GASTRIC CANCER

    by Mohammed Soutto, Xing Zhang, Zheng Chen, Nadeem S. Bhat, Shoumin Zhu, Heng Lu, Dunfa Peng, Zekuan Xu, Wael El-Rifai, Oliver G. McDonald, et al.

  • Open Access

    FeatherTTS: Robust and Efficient attention based Neural TTS

    by Qiao Tian, Zewang Zhang, Chao Liu, Heng Lu, Linghui Chen, Bin Wei, Pujiang He, Shan Liu

    Attention based neural TTS is elegant speech synthesis pipeline and has shown a powerful ability to generate natural speech. However, it is still not robust enough to meet the stability requirements for industrial products. Besides, it suffers from slow inference speed owning to the autoregressive generation process. In this work, we propose FeatherTTS, a robust and efficient attention-based neural TTS system. Firstly, we propose a novel Gaussian attention which utilizes interpretability of Gaussian attention and the strict monotonic property in TTS. By this method, we replace the commonly used stop token prediction architecture with attentive stop prediction. Secondly, we apply block sparsity on the autoregressive decoder to speed up speech synthesis. The experimental results show that our proposed FeatherTTS not only nearly eliminates the problem of word skipping, repeating in particularly hard texts and keep the naturalness of generated speech, but also speeds up acoustic feature generation by 3.5 times over Tacotron. Overall, the proposed FeatherTTS can be $35$x faster than real-time on a single CPU.