RSRongfeng Su

Rongfeng Su

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Rongfeng Su has a total of 45 co-authors. Here are the top co-authors by number of shared publications:

  • Lan Wang

    Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    6 shared publications
  • Nan Yan

    Chinese Academy of Sciences

    6 shared publications
  • Xunying Liu

    Chinese University of Hong Kong

    3 shared publications
  • Yudong Yang

    Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    3 shared publications
  • Juan Liu

    ICAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    2 shared publications
  • Manwa Lawrence Ng

    Speech Science Laboratory, Faculty of Education, University of Hong Kong, Hong Kong SAR, China

    2 shared publications

Latest Publications

  • Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detection

    by Rongfeng Su, Changqing Xu, Hangbin Yu, Xinyi Wu, Feng Xu, Xie Chen, Lan Wang, Nan Yan

    Previous studies have demonstrated that emotional features from a single acoustic sentiment label can enhance depression diagnosis accuracy. Additionally, according to the Emotion Context-Insensitivity theory and our pilot study, individuals with depression might convey negative emotional content in an unexpectedly calm manner, showing a high degree of inconsistency in emotional expressions during natural conversations. So far, few studies have recognized and leveraged the emotional expression inconsistency for depression detection. In this paper, a multimodal cross-attention method is presented to capture the Acoustic-Textual Emotional Inconsistency (ATEI) information. This is achieved by analyzing the intricate local and long-term dependencies of emotional expressions across acoustic and textual domains, as well as the mismatch between the emotional content within both domains. A Transformer-based model is then proposed to integrate this ATEI information with various fusion strategies for detecting depression. Furthermore, a scaling technique is employed to adjust the degree of each ATEI element in relation to the final depression diagnosis during the fusion process, thereby enhancing the model's ability to discern patients with depression across varying levels of severity. To best of our knowledge, this work is the first to incorporate emotional expression inconsistency information into depression detection. Experimental results on a counseling conversational dataset illustrate the effectiveness of our method.


  • Open Access

    An Audio-Ultrasound Synchronized Database of Tongue Movement for Mandarin speech

    by Yudong Yang, Rongfeng Su, Shaofeng Zhao, Jianguo Wei, Manwa Lawrence Ng, Nan Yan, Lan Wang

    Ultrasound imaging has been widely adopted in speech research to visualize dynamic tongue movements during speech production. These images are universally used as visual feedback in interventions for articulation disorders or visual cues in speech recognition. Nevertheless, the availability of high-quality audio-ultrasound datasets remains scarce. The present study, therefore, aims to construct a multimodal database designed for Mandarin speech. The dataset integrates synchronized ultrasound images of lingual movement, and the corresponding audio recordings and text annotations elicited from 43 healthy speakers and 11 patients with dysarthria through speech tasks (including vowels, monosyllables, and sentences), with a total duration of 22.31 hours. In addition, a customized helmet structure was employed to stabilize the ultrasound probe, precisely controlling for head movement and minimizing displacement interference, The proposed database carries apparent values in automatic speech recognition, silent interface development, and research in speech pathology and linguistics.


  • Automatic Assessment of Chinese Dysarthria Using Audio-visual Vowel Graph Attention Network

    by Xiaokang Liu, Xiaoxia Du, Juan Liu, Rongfeng Su, Manwa Lawrence Ng, Yumei Zhang, Yudong Yang, Shaofeng Zhao, Lan Wang, Nan Yan

    Automatic assessment of dysarthria remains a highly challenging task due to the high heterogeneity in acoustic signals and the limited data. Currently, research on the automatic assessment of dysarthria primarily focuses on two approaches: one that utilizes expert features combined with machine learning, and the other that employs data-driven deep learning methods to extract representations. Studies have shown that expert features can effectively account for the heterogeneity of dysarthria but may lack comprehensiveness. In contrast, deep learning methods excel at uncovering latent features. Therefore, integrating the advantages of expert knowledge and deep learning to construct a neural network architecture based on expert knowledge may be beneficial for interpretability and assessment performance. In this context, the present paper proposes a vowel graph attention network based on audio-visual information, which effectively integrates the strengths of expert knowledge and deep learning. Firstly, the VGAN (Vowel Graph Attention Network) structure based on vowel space theory was designed, which has two branches to mine the information in features and the spatial correlation between vowels respectively. Secondly, a feature set based on expert knowledge and deep representation is designed. Finally, visual information was incorporated into the model to further enhance its robustness and generalizability. Tested on the Mandarin Subacute Stroke Dysarthria Multimodal (MSDM) Database, this method exhibited superior performance in regression experiments targeting Frenchay scores compared to existing approaches.


  • A Transformer-Based Depression Detection Network Leveraging Speech Emotional Expression Cues

    by Changqing Xu, Xinyi Wu, Nan Li, Xin Wang, Xu Feng, Rongfeng Su, Nan Yan, Lan Wang


  • Feature Extraction Method Based on Contrastive Learning for Dysarthria Detection

    by Yudong Yang, Xinyi Wu, Xiaokang Liu, Juan Liu, Jingdong Zhou, Rennan Wang, Xin Wang, Rongfeng Su, Nan Yan, Lan Wang


  • Open Access

    A Speech-Based Mobile Screening Tool for Mild Cognitive Impairment: Technical Performance and User Engagement Evaluation

    by Rukiye Ruzi, Yue Pan, Menwa Lawrence Ng, Rongfeng Su, Lan Wang, Jianwu Dang, Liwei Liu, Nan Yan

    Traditional screening methods for Mild Cognitive Impairment (MCI) face limitations in accessibility and scalability. To address this, we developed and validated a speech-based automatic screening app implementing three speech–language tasks with user-centered design and server–client architecture. The app integrates automated speech processing and SVM classifiers for MCI detection. Functionality validation included comparison with manual assessment and testing in real-world settings (n = 12), with user engagement evaluated separately (n = 22). The app showed comparable performance with manual assessment (F1 = 0.93 vs. 0.95) and maintained reliability in real-world settings (F1 = 0.86). Task engagement significantly influenced speech patterns: users rating tasks as “most interesting” produced more speech content (p < 0.05), though behavioral observations showed consistent cognitive processing across perception groups. User engagement analysis revealed high technology acceptance (86%) across educational backgrounds, with daily cognitive exercise habits significantly predicting task benefit perception (H = 9.385, p < 0.01). Notably, perceived task difficulty showed no significant correlation with cognitive performance (p = 0.119), suggesting the system’s accessibility to users of varying abilities. While preliminary, the mobile app demonstrated both robust assessment capabilities and sustained user engagement, suggesting the potential viability of widespread cognitive screening in the geriatric population.


  • Open Access

    An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data

    by Yudong Yang, Rongfeng Su, Xiaokang Liu, Nan Yan, Lan Wang

    Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator movements, such as ultrasound tongue imaging (UTI) data. An issue of existing AAI methods is only using the personalized acoustic information to derive the general patterns of tongue motions, and thus the quality of generated UTI data is limited. To address this issue, this paper proposes an audio-textual diffusion model for the UTI data generation task. In this model, the inherent acoustic characteristics of individuals related to the tongue motion details are encoded by using wav2vec 2.0, while the ASR transcriptions related to the universality of tongue motions are encoded by using BERT. UTI data are then generated by using a diffusion module. Experimental results showed that the proposed diffusion model could generate high-quality UTI data with clear tongue contour that is crucial for the linguistic analysis and clinical assessment. The project can be found on the website\footnote{https://yangyudong2020.github.io/wav2uti/

  • Open Access

    On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition

    by Mengzhe Geng, Xurong Xie, Rongfeng Su, Jianwei Yu, Zengrui Jin, Tianzi Wang, Shujie Hu, Zi Ye, Helen Meng, Xunying Liu

    Accurate recognition of dysarthric and elderly speech remain challenging tasks to date. Speaker-level heterogeneity attributed to accent or gender, when aggregated with age and speech impairment, create large diversity among these speakers. Scarcity of speaker-level data limits the practical use of data-intensive model based speaker adaptation methods. To this end, this paper proposes two novel forms of data-efficient, feature-based on-the-fly speaker adaptation methods: variance-regularized spectral basis embedding (SVR) and spectral feature driven f-LHUC transforms. Experiments conducted on UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest the proposed on-the-fly speaker adaptation approaches consistently outperform baseline iVector adapted hybrid DNN/TDNN and E2E Conformer systems by statistically significant WER reduction of 2.48%-2.85% absolute (7.92%-8.06% relative), and offline model based LHUC adaptation by 1.82% absolute (5.63% relative) respectively.

  • Cross-Domain Deep Visual Feature Generation for Mandarin Audio–Visual Speech Recognition

    by Rongfeng Su, Xunying Liu, Lan Wang, Jingzhou Yang

    There has been a long term interest in using visual information to improve automatic speech recognition (ASR) system performance. Both audio and visual information are required in conventional audio visual speech recognition (AVSR) systems. This limits their wider applications when visual modality is not present. To this end, one possible solution is to use acousticto-visual (A2V) inversion techniques to generate visual features. Previous research in this direction used synthetic acousticarticulatory parallel data in inversion model training. The acoustic mismatch between the audio-visual (AV) parallel data and target data was not considered. In addition, the target language to apply these technologies has been focused on English. In this paper, a real 3D Audio-Visual Mandarin Continuous Speech (3DAV-MCS) corpus was used to train deep neural network based A2V inversion models. Cross-domain adaptation of the inversion models allows suitable visual features to be generated from acoustic data of mismatched domains. The proposed crossdomain deep visual feature generation techniques were evaluated on two state-of-the-art Mandarin speech recognition tasks: DAPRA GALE broadcast transcription and BOLT conversational telephone speech recognition. The AVSR systems constructed using the cross-domain generated visual features consistently outperformed the baseline convolutional neural network (CNN) ASR systems by up to 3.3% absolute (9.1% relative) character error rate (CER) reductions after both speaker adaptive training and sequence discriminative training were performed.