Zhe Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhe Li has a total of 126 co-authors. Here are the top co-authors by number of shared publications:
- Jiabao Sheng
Department of Pathology, The University of Hong Kong, Hong Kong, SAR, China
7 shared publications - Xiuhong Li
Xinjiang University
6 shared publications - Chenyu Zhou
Xinjiang University
5 shared publications - Man-Wai Mak
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, SAR, China
5 shared publications Fan Chen
Xinjiang University
4 shared publicationsJiabao Sheng
Stanford University
3 shared publications
Latest Publications
- 24 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 4697-4711
WhisMultiNet: Advancing End-to-End Speech Topic Classification With Whisper and MultiGateGNN
Speech topic classification aims to automatically identify and categorize the topics of audio content, with widespread applications in recommendation systems, speech content indexing, and other fields. In recent years, end-to-end models have gradually outperformed cascaded models due to their ability to preserve speech features and avoid ASR errors. However, existing end-to-end models still face challenges in deep semantic extraction and global semantic modeling. To address these issues, we propose WhisMultiNet, an end-to-end speech topic classification model based on Whisper, a feature projector, multi-scale CNNs, and a multi-layer graph network. Specifically, Whisper extracts deep speech features from the last layer of its encoder, while the pre-trained feature projector enhances the semantic representation of speech features. The multi-scale CNNs further improve the multi-level representation of local semantic features. A multi-layer graph network constructs nodes by segmenting semantic feature sequences and employs a dynamic gating mechanism to weight and fuse graph attention outputs from different layers, effectively capturing global semantic information. Our model outperforms previous state-of-the-art models, achieving accuracy improvements of 4.01% and 7.23% on the Fisher and Switchboard datasets, respectively. Furthermore, to address the limited diversity and accessibility of existing datasets for speech topic classification, we crawled, filtered, and curated a 638-hour multimodal speech topic classification dataset covering 40 categories, named TedSpeechTopic. This dataset will be publicly released alongside the paper at the following link.
- 24 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 4697-4711
WhisMultiNet: Advancing End-to-End Speech Topic Classification With Whisper and MultiGateGNN
Speech topic classification aims to automatically identify and categorize the topics of audio content, with widespread applications in recommendation systems, speech content indexing, and other fields. In recent years, end-to-end models have gradually outperformed cascaded models due to their ability to preserve speech features and avoid ASR errors. However, existing end-to-end models still face challenges in deep semantic extraction and global semantic modeling. To address these issues, we propose WhisMultiNet, an end-to-end speech topic classification model based on Whisper, a feature projector, multi-scale CNNs, and a multi-layer graph network. Specifically, Whisper extracts deep speech features from the last layer of its encoder, while the pre-trained feature projector enhances the semantic representation of speech features. The multi-scale CNNs further improve the multi-level representation of local semantic features. A multi-layer graph network constructs nodes by segmenting semantic feature sequences and employs a dynamic gating mechanism to weight and fuse graph attention outputs from different layers, effectively capturing global semantic information. Our model outperforms previous state-of-the-art models, achieving accuracy improvements of 4.01% and 7.23% on the Fisher and Switchboard datasets, respectively. Furthermore, to address the limited diversity and accessibility of existing datasets for speech topic classification, we crawled, filtered, and curated a 638-hour multimodal speech topic classification dataset covering 40 categories, named TedSpeechTopic. This dataset will be publicly released alongside the paper at the following link.
- 15 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 3896-3907
Disentangling Speech Representations Learning With Latent Diffusion for Speaker Verification
Disentangled speech representation learning for speaker verification aims to separate spoken content and speaker timbre into distinct representations. However, existing variational autoencoder (VAE)–based methods for speech disentanglement rely on latent variables that lack semantic meaning, limiting their effectiveness for speaker verification. To address this limitation, we propose a diffusion-based method that disentangles and separates speaker features and speech content in the latent space. Building upon the VAE framework, we employ a speaker encoder to learn latent variables representing speaker features while using frame-specific latent variables to capture content. Unlike previous sequential VAE approaches, our method utilizes a conditional diffusion model in the latent space to derive speaker-aware representations. Experiments on the VoxCeleb and CN-Celeb datasets demonstrate that our method effectively isolates speaker features from speech content using pre-trained speech representations. The learned embeddings are robust to language mismatches since the speaker embeddings become content-invariant after content removal. Additionally, we design contrastive learning experiments showing that our training objective can enhance the learning of speaker-discriminative embeddings without relying on classification-based loss.
- 15 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 3896-3907
Disentangling Speech Representations Learning With Latent Diffusion for Speaker Verification
Disentangled speech representation learning for speaker verification aims to separate spoken content and speaker timbre into distinct representations. However, existing variational autoencoder (VAE)–based methods for speech disentanglement rely on latent variables that lack semantic meaning, limiting their effectiveness for speaker verification. To address this limitation, we propose a diffusion-based method that disentangles and separates speaker features and speech content in the latent space. Building upon the VAE framework, we employ a speaker encoder to learn latent variables representing speaker features while using frame-specific latent variables to capture content. Unlike previous sequential VAE approaches, our method utilizes a conditional diffusion model in the latent space to derive speaker-aware representations. Experiments on the VoxCeleb and CN-Celeb datasets demonstrate that our method effectively isolates speaker features from speech content using pre-trained speech representations. The learned embeddings are robust to language mismatches since the speaker embeddings become content-invariant after content removal. Additionally, we design contrastive learning experiments showing that our training objective can enhance the learning of speaker-discriminative embeddings without relying on classification-based loss.
- 26 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2961-2972
Mutual Information-Enhanced Contrastive Learning With Margin for Maximal Speaker Separability
Contrastive learning across various augmentations of the same utterance can enhance speaker representations' ability to distinguish new speakers. This paper introduces a supervised contrastive learning objective that optimizes a speaker embedding space using label information from training data. Besides augmenting different segments of an utterance to form a positive pair, our approach generates multiple positive pairs by augmenting various utterances from the same speaker. However, employing contrastive learning for speaker verification (SV) presents two challenges: (1) softmax loss is ineffective in reducing intra-class variation, and (2) previous research has shown that contrastive learning can share information across the augmented views of an object but could discard task-relevant nonshared information, suggesting that it is essential to keep nonshared speaker information across the augmented views when constructing a speaker representation space. To overcome the first challenge, we incorporate an additive angular margin in the contrastive loss. For the second challenge, we maximize the mutual information (MI) between the squeezed low-level features and speaker representations to extract the nonshared information. Evaluations on VoxCeleb, CN-Celeb, and CU-MARVEL validate that our new learning objective enables ECAPA-TDNN to identify an embedding space that exhibits robust speaker discrimination.
- 26 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2961-2972
Mutual Information-Enhanced Contrastive Learning With Margin for Maximal Speaker Separability
Contrastive learning across various augmentations of the same utterance can enhance speaker representations' ability to distinguish new speakers. This paper introduces a supervised contrastive learning objective that optimizes a speaker embedding space using label information from training data. Besides augmenting different segments of an utterance to form a positive pair, our approach generates multiple positive pairs by augmenting various utterances from the same speaker. However, employing contrastive learning for speaker verification (SV) presents two challenges: (1) softmax loss is ineffective in reducing intra-class variation, and (2) previous research has shown that contrastive learning can share information across the augmented views of an object but could discard task-relevant nonshared information, suggesting that it is essential to keep nonshared speaker information across the augmented views when constructing a speaker representation space. To overcome the first challenge, we incorporate an additive angular margin in the contrastive loss. For the second challenge, we maximize the mutual information (MI) between the squeezed low-level features and speaker representations to extract the nonshared information. Evaluations on VoxCeleb, CN-Celeb, and CU-MARVEL validate that our new learning objective enables ECAPA-TDNN to identify an embedding space that exhibits robust speaker discrimination.
- 12 November 2024
- Published by Springer Nature
- p. 430-442
Enhancing Multimodal Rumor Detection with Statistical Image Features and Modal Alignment via Contrastive Learning
- 12 November 2024
- Published by Springer Nature
- p. 430-442
Enhancing Multimodal Rumor Detection with Statistical Image Features and Modal Alignment via Contrastive Learning
- 20 October 2024
- Published by Springer Nature
- p. 541-554
Enhancing Cross-Modal Alignment in Multimodal Sentiment Analysis via Prompt Learning
- 20 October 2024
- Published by Springer Nature
- p. 541-554
Enhancing Cross-Modal Alignment in Multimodal Sentiment Analysis via Prompt Learning
- 29 July 2024
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 84 (19) , 21117-21139
Locate, enhance and fuse: a progressively optimized network for camouflaged object detection
- 29 July 2024
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 84 (19) , 21117-21139
Locate, enhance and fuse: a progressively optimized network for camouflaged object detection
- 22 April 2024
- Published by Springer Nature in Neural Computing and Applications
- Vol. 36 (22) , 13513-13525
CEA-Net: a co-interactive external attention network for joint intent detection and slot filling
- 22 April 2024
- Published by Springer Nature in Neural Computing and Applications
- Vol. 36 (22) , 13513-13525
CEA-Net: a co-interactive external attention network for joint intent detection and slot filling