HMHelen Meng

Helen Meng

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Helen Meng has a total of 393 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • MeMo: Attentional Momentum for Real-time Audio-visual Target Speaker Extraction under Impaired Visual Conditions

    by Junjie Li, Wenxuan Wu, Shuai Wang, Zexu Pan, Kong Aik Lee, Helen Meng, Haizhou Li

    Audio-visual target speaker extraction (AV-TSE) faces significant performance degradation when visual cues are missing or impaired, limiting its practical deployment in real-world scenarios. This paper introduces MeMo, a novel framework that achieves attentional momentum through adaptive memory banks to maintain consistent focus on target speakers despite visual impairments. MeMo employs two complementary memory banks: a speaker bank storing identity features and a contextual bank capturing temporal speech patterns. The framework leverages self-enrolled speech from previous processing windows as supplementary reference, enabling robust extraction even when current visual cues are unreliable. Extensive experiments on the VoxCeleb2 dataset demonstrate MeMo's effectiveness, yielding approximately 27% relative improvement over the baseline model under impaired visual conditions in online streaming scenarios. The contextual bank proves particularly effective, providing over 2 dB gains across multiple evaluation metrics while maintaining real-time processing capabilities.

  • StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing

    by Liyang Chen, Tianze Zhou, Xu He, Boshi Tang, Zhiyong Wu, Yang Huang, Yang Wu, Zhongqian Sun, Wei Yang, Helen Meng

    The visual dubbing task aims to generate mouth movements synchronized with the driving audio, which has seen significant progress in recent years. However, two critical deficiencies hinder their wide application: (1) Audio-only driving paradigms inadequately capture speaker-specific lip habits, which fail to generate lip movements similar to the target avatar; (2) Conventional blind-inpainting approaches frequently produce visual artifacts when handling obstructions (e.g., microphones, hands), limiting practical deployment. In this paper, we propose StableDub, a novel and concise framework integrating lip-habit-aware modeling with occlusion-robust synthesis. Specifically, building upon the Stable-Diffusion backbone, we develop a lip-habit-modulated mechanism that jointly models phonemic audio-visual synchronization and speaker-specific orofacial dynamics. To achieve plausible lip geometries and object appearances under occlusion, we introduce the occlusion-aware training strategy by explicitly exposing the occlusion objects to the inpainting process. By incorporating the proposed designs, the model eliminates the necessity for cost-intensive priors in previous methods, thereby exhibiting superior training efficiency on the computationally intensive diffusion-based backbone. To further optimize training efficiency from the perspective of model architecture, we introduce a hybrid Mamba-Transformer architecture, which demonstrates the enhanced applicability in low-resource research scenarios. Extensive experimental results demonstrate that StableDub achieves superior performance in lip habit resemblance and occlusion robustness. Our method also surpasses other methods in audio-lip sync, video quality, and resolution consistency. We expand the applicability of visual dubbing methods from comprehensive aspects.

  • ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction

    by Wenxuan Wu, Shuai Wang, Xixin Wu, Helen Meng, Haizhou Li

    Audio-visual target speaker extraction (AV-TSE) models primarily rely on visual cues from the target speaker. However, humans also leverage linguistic knowledge, such as syntactic constraints, next word prediction, and prior knowledge of conversation, to extract target speech. Inspired by this observation, we propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models through three distinct guidance strategies: output linguistic constraints, intermediate linguistic prediction, and input linguistic prior. Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones demonstrate the effectiveness of our approach. Significant improvements are observed in challenging scenarios, including visual cue impaired, unseen languages, target speaker switches, increased interfering speakers, and out-of-domain test set.

  • Open Access

    Can Large Language Models Be Query Optimizer for Relational Databases?

    by Jie Tan, Kangfei Zhao, Rui Li, Jeffrey Xu Yu, Chengzhi Piao, Hong Cheng, Helen Meng, Deli Zhao, Yu Rong

    Query optimization is a complex planning and decision-making problem within the exponentially growing plan space in database management systems (DBMS). Traditional optimization techniques have been extensively studied over decades, leaving limited room for further improvement along this track. Recent developments of Large Language Models (LLMs) have demonstrated their potential in solving complex planning and decision-making problems, such as arithmetic and programmatic tasks. In this paper, we try to explore the potential of LLMs in handling query optimization and propose a tentative LLM-based query optimizer dubbed LLM-QO, established on PostgreSQL's execution engine. In LLM-QO, we formulate query optimization in an autoregressive fashion which directly generates the execution plan without explicit plan enumeration. To investigate the essential input of LLM-QO, we design a customized data recipe named QInstruct to collect the training data from various optimizers and serialize the database's meta data, queries and corresponding plans into a textual format. Based on QInstruct, we implement a two-stage fine-tuning pipeline, Query Instruction Tuning (QIT) and Query Direct Preference Optimization (QDPO), to empower the capability of general-purpose LLMs in handling query optimization. In our experiments, LLM-QO can generate valid and high-quality plans and consistently outperforms both traditional and learned optimizers on three query workloads. Our findings verify that LLMs can be derived as query optimizers where generalization, efficiency and adaptivity deserve further research efforts.


  • PerTTS: Personalized and Controllable Zero-Shot Spontaneous Style Text-to-Speech Synthesis

    by Weiqin Li, Qian Chen, Dan Luo, Tianjiao Du, Yafeng Chen, Zhiyong Wu, Xixin Wu, Helen Meng

    In spoken scenarios, achieving personalized and controllable zero-shot spontaneous style speech synthesis is highly significant, particularly in generating natural and expressive speech for unseen speakers under data-limited conditions. Traditional methods typically achieve this by fine-tuning pre-trained multi-speaker speech synthesis models or adopting zero-shot adaptation techniques. However, these methods exhibit limitations in voice cloning and style modeling, struggling to capture f ine-grained voice characteristics and complex speaking styles of target speakers. In this paper, we propose PerTTS, a personalized and controllable zero-shot spontaneous speech synthesis method. This approach introduces a personalized speaking style encoder that utilizes pre-trained models and a local prosody encoder to extract semantic, duration, timbre and prosody information from multiple reference utterances of the target speaker, thereby forming a comprehensive personalized representation of speaking style. Furthermore, we employ knowledge distillation to learn spontaneous behavior patterns and incorporate a multi-modal pseudo label detector to extract labels from unlabeled data, enabling modeling and control of spontaneous behaviors. This mechanism significantly enhances the naturalness and spontaneity of the synthesized speech. Experimental results demonstrate that PerTTS significantly outperforms existing models in terms of speaking style similarity and speech naturalness. The introduction of personalized speaking style representations effectively improves style similarity, and the incorporation of spontaneous behavior modeling further improves the naturalness and spontaneity of the synthesized speech, while enabling controllable generation of spontaneous behaviors.

  • Guest Editorial: Modeling and Processing Language and Speech in Neurodegenerative Disorders

    by Juan I. Godino-Llorente, Julián D. Arias Londoño, Mark Hasegawa-Johnson, Helen Meng, Stefanie Shattuck-Hufnagel

  • Open Access

    Towards Effective and Efficient Non-Autoregressive Decoders for Conformer and LLM-Based ASR Using Block-Based Attention Mask

    by Tianzi Wang, Xurong Xie, Zengrui Jin, Mengzhe Geng, Jiajun Deng, Zhaoqing Li, Shoukang Hu, Shujie Hu, Guinan Li, Mingyu Cui, et al.

    Automatic speech recognition (ASR) systems often rely on autoregressive (AR) Transformer decoder architectures, which limit efficient inference parallelization due to their sequential nature. To this end, non-autoregressive (NAR) approaches aim primarily to achieve significant decoding speedup while the maintaining recognition accuracy that is comparable to AR baselines. This paper proposes a novel NAR block-based attention mask decoder (AMD) that effectively improves decoding efficiency while maintaining ASR accuracy, and also offering flexibility in balancing the performance-efficiency trade-off on both Conformer and large language model (LLM)-based ASR systems. The proposed AMD performs parallel inference within contiguous blocks of output labels while maintaining monotonic left-to-right prediction between blocks. A one-pass beam search algorithm is designed to dynamically fuse Connectionist Temporal Classification (CTC), AR decoder, and AMD probabilities. Experiments are conducted on normal speech LS960 and DBank elderly speech across: a) The Conformer encoder-decoder ASR system with filterbank input features; b) its integration with WavLM features; and c) further advancement by integrating an LLM-based decoder. On the LS960 task, the proposed AMD empowered tripartite decoder achieves decoding speedup ratios of up to 1.44x, 1.55x, and 2.31x under the three model configurations over the CTC + AR baselines, without statistically significant WER increases. When operating with real-time factors (RTFs) comparable to the baselines, the tripartite decoder produces statistically significant WER reductions of 0.19%, 0.62% and 0.13% absolute (4.3%, 16.3%, and 3.8% relative). Similar improvements are also obtained on the DBank task, where the tripartite decoder accelerates decoding by up to 1.38x, 1.64x and 1.61x without statistically significant WER increase, and yields statistically significant WER reductions of 0.46%, 0.38% and 0.41% absolute (1.8%, 1.8%, and 2.0% relative) when operating with RTFs comparable to the CTC + AR baselines.


  • Detecting Neurocognitive Disorders Through Analyses of Topic Evolution and Cross-Modal Consistency in Visual-Stimulated Narratives

    by Jinchao Li, Yuejiao Wang, Junan Li, Jiawen Kang, Bo Zheng, Ka Ho Wong, Brian Kan-Wing Mak, Helene H. Fung, Jean Woo, Man-Wai Mak, et al.

    Early detection of neurocognitive disorders (NCDs) is crucial for timely intervention and disease management. Given that language impairments manifest early in NCD progression, visual-stimulated narrative (VSN)-based analysis offers a promising avenue for NCD detection. Current VSN-based NCD detection methods primarily focus on linguistic microstructures (e.g., lexical diversity) that are closely tied to bottom-up, stimulus-driven cognitive processes. While these features illuminate basic language abilities, the higher-order linguistic macrostructures (e.g., topic development) that may reflect top-down, concept-driven cognitive abilities remain underexplored. These macrostructural patterns are crucial for NCD detection, yet challenging to quantify due to their abstract and complex nature. To bridge this gap, we propose two novel macrostructural approaches: (1) a Dynamic Topic Model (DTM) to track topic evolution over time, and (2) a Text-Image Temporal Alignment Network (TITAN) to measure cross-modal consistency between narrative and visual stimuli. Experimental results show the effectiveness of the proposed approaches in NCD detection, with TITAN achieving superior performance across three corpora: ADReSS (F1=0.8889), ADReSSo (F1=0.8504), and CU-MARVEL-RABBIT (F1=0.7238). Feature contribution analysis reveals that macrostructural features (e.g., topic variability, topic change rate, and topic consistency) constitute the most significant contributors to the model's decision pathways, outperforming the investigated microstructural features. These findings underscore the value of macrostructural analysis for understanding linguistic-cognitive interactions associated with NCDs.