Ziyang Ma
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Ziyang Ma has a total of 64 co-authors. Here are the top co-authors by number of shared publications:
- Xie Chen
Shanghai Jiao Tong University, Shanghai Innovation Institute, Shanghai, China
6 shared publications - Kai Yu
Shanghai Jiao Tong University, Shanghai, China
3 shared publications - Zhikang Niu
Shanghai Jiao Tong University, Shanghai, China
3 shared publications - Chen Yang
Shanghai Jiao Tong University, Shanghai, China
2 shared publications - Shiliang Zhang
Tongyi Lab, Alibaba Group, Hangzhou, China
2 shared publications - Wenxi Chen
Shanghai Jiao Tong University, Shanghai, China
2 shared publications
Latest Publications
- Open Access
- 11 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 128-130
A Diagnostic Evaluation of End-to-End Spoken Dialogue Models via URO-Bench
End-to-end spoken dialogue models (SDMs) turn text-based language models into real-time speech-to-speech systems. This transition raises a direct question: after speech adaptation, which language-model capabilities are retained, and which speech-specific abilities are gained? We revisit this question through URO-Bench, a comprehensive benchmark for evaluating SDMs across understanding, reasoning, and oral conversation. Results show that current open-source SDMs can handle everyday spoken conversations, but still lag behind cascaded or proprietary systems in instruction following and reasoning. At the same time, the pro-track results indicate remaining weaknesses in speech-specific interaction abilities, including speaker awareness, cross-lingual oral interaction, and acoustically grounded understanding. These findings suggest that future S2S systems should be evaluated by both cognitive capability retention and speech-specific capability acquisition.
- 11 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 128-130
Open AccessA Diagnostic Evaluation of End-to-End Spoken Dialogue Models via URO-Bench
End-to-end spoken dialogue models (SDMs) turn text-based language models into real-time speech-to-speech systems. This transition raises a direct question: after speech adaptation, which language-model capabilities are retained, and which speech-specific abilities are gained? We revisit this question through URO-Bench, a comprehensive benchmark for evaluating SDMs across understanding, reasoning, and oral conversation. Results show that current open-source SDMs can handle everyday spoken conversations, but still lag behind cascaded or proprietary systems in instruction following and reasoning. At the same time, the pro-track results indicate remaining weaknesses in speech-specific interaction abilities, including speaker awareness, cross-lingual oral interaction, and acoustically grounded understanding. These findings suggest that future S2S systems should be evaluated by both cognitive capability retention and speech-specific capability acquisition.
- 23 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2763-2773
Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
The growing reliance on large-scale speech data has made privacy protection a critical concern. However, existing anonymization approaches often degrade data utility, for example by disrupting acoustic continuity or reducing vocal diversity, which compromises the value of speech data for downstream tasks such as Automatic Speech Recognition (ASR), Text-to- Speech (TTS), and Speech Emotion Recognition (SER). Current evaluation practices are also limited, as they mainly rely on direct testing of anonymized speech with pretrained models, providing only a partial view of utility. To address these issues, we propose a novel two-stage framework that protects both linguistic content and acoustic identity while maintaining usability. For content privacy, we employ a generative speech editing model to seamlessly replace personally identifiable information (PII), and for voice privacy, we introduce F3-VA, a flow-matchingbased anonymization framework with a three-stage design that produces diverse and distinct anonymized speakers. To enable a more comprehensive assessment, we evaluate privacy using both acoustic- and content-based speaker verification metrics, and assess utility by training ASR, TTS, and SER models from scratch. Experimental results show that our framework achieves stronger privacy protection with minimal utility degradation compared to baselines from the VoicePrivacy Challenge, while the proposed evaluation protocol provides a more realistic reflection of the utility of anonymized speech under privacy protection.
- 23 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2763-2773
Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
The growing reliance on large-scale speech data has made privacy protection a critical concern. However, existing anonymization approaches often degrade data utility, for example by disrupting acoustic continuity or reducing vocal diversity, which compromises the value of speech data for downstream tasks such as Automatic Speech Recognition (ASR), Text-to- Speech (TTS), and Speech Emotion Recognition (SER). Current evaluation practices are also limited, as they mainly rely on direct testing of anonymized speech with pretrained models, providing only a partial view of utility. To address these issues, we propose a novel two-stage framework that protects both linguistic content and acoustic identity while maintaining usability. For content privacy, we employ a generative speech editing model to seamlessly replace personally identifiable information (PII), and for voice privacy, we introduce F3-VA, a flow-matchingbased anonymization framework with a three-stage design that produces diverse and distinct anonymized speakers. To enable a more comprehensive assessment, we evaluate privacy using both acoustic- and content-based speaker verification metrics, and assess utility by training ASR, TTS, and SER models from scratch. Experimental results show that our framework achieves stronger privacy protection with minimal utility degradation compared to baselines from the VoicePrivacy Challenge, while the proposed evaluation protocol provides a more realistic reflection of the utility of anonymized speech under privacy protection.
- Open Access
- 12 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 20 (1) , 63-76
SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.
- 12 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 20 (1) , 63-76
Open AccessSLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 10748-10757
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 10748-10757
Open AccessEmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
- Top Cited PapersOpen Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 13837-13842
MER 2025: When Affective Computing Meets Large Language Models
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 13837-13842
Top Cited PapersOpen AccessMER 2025: When Affective Computing Meets Large Language Models