Xingyu Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Xingyu Zhang has a total of 53 co-authors. Here are the top co-authors by number of shared publications:
- Erwei Yin
Defense Innovation Institute, Academy of Military Sciences, Beijing, China
10 shared publications - Liang Xie
Defense Innovation Institute, Academy of Military Sciences, Beijing, China
9 shared publications - Changyan Zheng
Defense Innovation Institute, Academy of Military Sciences, Beijing 100071, China
8 shared publications - Yakun Zhang
Defense Innovation Institute, Academy of Military Sciences, Beijing, China
6 shared publications - Meng Sun
Peking University, Beijing, China
3 shared publications Xia Zou
3 shared publications
Latest Publications
- 9 May 2026
- Published by Elsevier in Biocybernetics and Biomedical Engineering
- Vol. 46 (2) , 432-454
Silent speech recognition and interaction via surface electromyography: A review
- 9 May 2026
- Published by Elsevier in Biocybernetics and Biomedical Engineering
- Vol. 46 (2) , 432-454
Silent speech recognition and interaction via surface electromyography: A review
- Open Access
- 26 February 2026
- Published by Springer Nature in Journal of King Saud University Computer and Information Sciences
- Vol. 38 (4)
VSMatch-Lip: a visual-semantic matching framework for zero-shot lip reading
Lip reading interprets speech from visual lip movements, offering a vital complement to audio-based recognition in challenging acoustic environments. However, existing models rely on supervised, closed-set classification and fail to recognize out-of-vocabulary words, severely limiting their practical application. To address this zero-shot learning (ZSL) challenge, we propose VSMatch-Lip, a non-generative visual-semantic matching framework. Our approach is grounded in the insight that while lip movements represent a visual manifestation of phonetics, providing a strong physical correlation for generalization, relying solely on this correlation is insufficient due to ambiguities like homophones. Therefore, our core innovation lies in introducing a multi-source fused semantic representation that synergistically integrates lexical meaning with powerful phonetic cues. This design allows the phonetic component to ground the alignment in visual articulation, while the semantic component provides crucial disambiguation, creating a more robust and discriminative target for matching. To effectively optimize this matching process, we design a tailored contrastive learning framework with specialized optimization strategies to tackle the large intra-class variance and training instability. As a key contribution, we also establish the first comprehensive ZSL benchmark on large-scale, in-the-wild datasets. Extensive experiments on this benchmark demonstrate that VSMatch-Lip achieves state-of-the-art performance, consistently outperforming all baselines, including contemporary generative models. Notably, under a 19:1 seen-to-unseen ratio on LRW, it surpasses the strongest generative baseline by nearly 9% in Top-1 unseen accuracy. To the best of our knowledge, this is the first successful and rigorous validation of a non-generative, direct matching ZSL framework on large-scale, in-the-wild lip reading benchmarks.
- 26 February 2026
- Published by Springer Nature in Journal of King Saud University Computer and Information Sciences
- Vol. 38 (4)
Open AccessVSMatch-Lip: a visual-semantic matching framework for zero-shot lip reading
Lip reading interprets speech from visual lip movements, offering a vital complement to audio-based recognition in challenging acoustic environments. However, existing models rely on supervised, closed-set classification and fail to recognize out-of-vocabulary words, severely limiting their practical application. To address this zero-shot learning (ZSL) challenge, we propose VSMatch-Lip, a non-generative visual-semantic matching framework. Our approach is grounded in the insight that while lip movements represent a visual manifestation of phonetics, providing a strong physical correlation for generalization, relying solely on this correlation is insufficient due to ambiguities like homophones. Therefore, our core innovation lies in introducing a multi-source fused semantic representation that synergistically integrates lexical meaning with powerful phonetic cues. This design allows the phonetic component to ground the alignment in visual articulation, while the semantic component provides crucial disambiguation, creating a more robust and discriminative target for matching. To effectively optimize this matching process, we design a tailored contrastive learning framework with specialized optimization strategies to tackle the large intra-class variance and training instability. As a key contribution, we also establish the first comprehensive ZSL benchmark on large-scale, in-the-wild datasets. Extensive experiments on this benchmark demonstrate that VSMatch-Lip achieves state-of-the-art performance, consistently outperforming all baselines, including contemporary generative models. Notably, under a 19:1 seen-to-unseen ratio on LRW, it surpasses the strongest generative baseline by nearly 9% in Top-1 unseen accuracy. To the best of our knowledge, this is the first successful and rigorous validation of a non-generative, direct matching ZSL framework on large-scale, in-the-wild lip reading benchmarks.
- 11 January 2026
- Published by Elsevier in Neural Networks
- Vol. 198, 108565
Sequential viseme-driven visual speech recognition through dual-stream interactive neural architecture
- 11 January 2026
- Published by Elsevier in Neural Networks
- Vol. 198, 108565
Sequential viseme-driven visual speech recognition through dual-stream interactive neural architecture
- 29 October 2025
- Published by Elsevier in Knowledge-Based Systems
- Vol. 331
Bridging semantics across modalities: Decoupled representation learning for audio-visual speech recognition
- 29 October 2025
- Published by Elsevier in Knowledge-Based Systems
- Vol. 331
Bridging semantics across modalities: Decoupled representation learning for audio-visual speech recognition
- 2 July 2025
- Published by Elsevier in Neural Networks
- Vol. 191, 107783
MsDUNE: A multi-scale masked temporal fusion framework for speaker-independent lipreading via Dirichlet uncertainty estimation
- 2 July 2025
- Published by Elsevier in Neural Networks
- Vol. 191, 107783
MsDUNE: A multi-scale masked temporal fusion framework for speaker-independent lipreading via Dirichlet uncertainty estimation
- 14 June 2025
- Published by Springer Nature
- p. 268-282
Advancing Lip-Reading for Unseen Speakers Through Fusion and Augmentation of Spatio-temporal Landmarks and Visual Features
- 14 June 2025
- Published by Springer Nature
- p. 268-282
Advancing Lip-Reading for Unseen Speakers Through Fusion and Augmentation of Spatio-temporal Landmarks and Visual Features
- Open Access
LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition
Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions.
Open AccessLipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition
Visual speech recognition (VSR), commonly known as lip reading, has garnered significant attention due to its wide-ranging practical applications. The advent of deep learning techniques and advancements in hardware capabilities have significantly enhanced the performance of lip reading models. Despite these advancements, existing datasets predominantly feature stable video recordings with limited variability in lip movements. This limitation results in models that are highly sensitive to variations encountered in real-world scenarios. To address this issue, we propose a novel framework, LipGen, which aims to improve model robustness by leveraging speech-driven synthetic visual data, thereby mitigating the constraints of current datasets. Additionally, we introduce an auxiliary task that incorporates viseme classification alongside attention mechanisms. This approach facilitates the efficient integration of temporal information, directing the model's focus toward the relevant segments of speech, thereby enhancing discriminative capabilities. Our method demonstrates superior performance compared to the current state-of-the-art on the lip reading in the wild (LRW) dataset and exhibits even more pronounced advantages under challenging conditions.
- 26 December 2024
- Published by Springer Nature in Applied Intelligence
- Vol. 55 (3) , 1-19
DuAGNet: an unrestricted multimodal speech recognition framework using dual adaptive gating fusion
- 26 December 2024
- Published by Springer Nature in Applied Intelligence
- Vol. 55 (3) , 1-19
DuAGNet: an unrestricted multimodal speech recognition framework using dual adaptive gating fusion
- 3 September 2024
- Published by IOP Publishing in Journal of Neural Engineering
- Vol. 21 (5) , 056001
A simplified adversarial architecture for cross-subject silent speech recognition using electromyography
Objective. The decline in the performance of electromyography (EMG)-based silent speech recognition is widely attributed to disparities in speech patterns, articulation habits, and individual physiology among speakers. Feature alignment by learning a discriminative network that resolves domain offsets across speakers is an effective method to address this problem. The prevailing adversarial network with a branching discriminator specializing in domain discrimination renders insufficiently direct contribution to categorical predictions of the classifier. Approach. To this end, we propose a simplified discrepancy-based adversarial network with a streamlined end-to-end structure for EMG-based cross-subject silent speech recognition. Highly aligned features across subjects are obtained by introducing a Nuclear-norm Wasserstein discrepancy metric on the back end of the classification network, which could be utilized for both classification and domain discrimination. Given the low-level and implicitly noisy nature of myoelectric signals, we devise a cascaded adaptive rectification network as the front-end feature extraction network, adaptively reshaping the intermediate feature map with automatically learnable channel-wise thresholds. The resulting features effectively filter out domain-specific information between subjects while retaining domain-invariant features critical for cross-subject recognition. Main results. A series of sentence-level classification experiments with 100 Chinese sentences demonstrate the efficacy of our method, achieving an average accuracy of 89.46% tested on 40 new subjects by training with data from 60 subjects. Especially, our method achieves a remarkable 10.07% improvement compared to the state-of-the-art model when tested on 10 new subjects with 20 subjects employed for training, surpassing its result even with three times training subjects. Significance. Our study demonstrates an improved classification performance of the proposed adversarial architecture using cross-subject myoelectric signals, providing a promising prospect for EMG-based speech interactive application.
- 3 September 2024
- Published by IOP Publishing in Journal of Neural Engineering
- Vol. 21 (5) , 056001
A simplified adversarial architecture for cross-subject silent speech recognition using electromyography
Objective. The decline in the performance of electromyography (EMG)-based silent speech recognition is widely attributed to disparities in speech patterns, articulation habits, and individual physiology among speakers. Feature alignment by learning a discriminative network that resolves domain offsets across speakers is an effective method to address this problem. The prevailing adversarial network with a branching discriminator specializing in domain discrimination renders insufficiently direct contribution to categorical predictions of the classifier. Approach. To this end, we propose a simplified discrepancy-based adversarial network with a streamlined end-to-end structure for EMG-based cross-subject silent speech recognition. Highly aligned features across subjects are obtained by introducing a Nuclear-norm Wasserstein discrepancy metric on the back end of the classification network, which could be utilized for both classification and domain discrimination. Given the low-level and implicitly noisy nature of myoelectric signals, we devise a cascaded adaptive rectification network as the front-end feature extraction network, adaptively reshaping the intermediate feature map with automatically learnable channel-wise thresholds. The resulting features effectively filter out domain-specific information between subjects while retaining domain-invariant features critical for cross-subject recognition. Main results. A series of sentence-level classification experiments with 100 Chinese sentences demonstrate the efficacy of our method, achieving an average accuracy of 89.46% tested on 40 new subjects by training with data from 60 subjects. Especially, our method achieves a remarkable 10.07% improvement compared to the state-of-the-art model when tested on 10 new subjects with 20 subjects employed for training, surpassing its result even with three times training subjects. Significance. Our study demonstrates an improved classification performance of the proposed adversarial architecture using cross-subject myoelectric signals, providing a promising prospect for EMG-based speech interactive application.