Zhikang Niu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Zhikang Niu has a total of 42 co-authors. Here are the top co-authors by number of shared publications:
- Xie Chen
Shanghai Jiao Tong University, Shanghai Innovation Institute, Shanghai, China
4 shared publications - Ziyang Ma
Shanghai Jiao Tong University, Shanghai, China
3 shared publications - Chen Yang
Shanghai Jiao Tong University, Shanghai, China
2 shared publications Guanrou Yang
X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence Shanghai Jiao Tong University, Shanghai, China
2 shared publications- Kai Yu
Shanghai Jiao Tong University, Shanghai, China
2 shared publications - Shiliang Zhang
Tongyi Lab, Alibaba Group, Hangzhou, China
2 shared publications
Latest Publications
- Open Access
- 11 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 128-130
A Diagnostic Evaluation of End-to-End Spoken Dialogue Models via URO-Bench
End-to-end spoken dialogue models (SDMs) turn text-based language models into real-time speech-to-speech systems. This transition raises a direct question: after speech adaptation, which language-model capabilities are retained, and which speech-specific abilities are gained? We revisit this question through URO-Bench, a comprehensive benchmark for evaluating SDMs across understanding, reasoning, and oral conversation. Results show that current open-source SDMs can handle everyday spoken conversations, but still lag behind cascaded or proprietary systems in instruction following and reasoning. At the same time, the pro-track results indicate remaining weaknesses in speech-specific interaction abilities, including speaker awareness, cross-lingual oral interaction, and acoustically grounded understanding. These findings suggest that future S2S systems should be evaluated by both cognitive capability retention and speech-specific capability acquisition.
- 11 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 128-130
Open AccessA Diagnostic Evaluation of End-to-End Spoken Dialogue Models via URO-Bench
End-to-end spoken dialogue models (SDMs) turn text-based language models into real-time speech-to-speech systems. This transition raises a direct question: after speech adaptation, which language-model capabilities are retained, and which speech-specific abilities are gained? We revisit this question through URO-Bench, a comprehensive benchmark for evaluating SDMs across understanding, reasoning, and oral conversation. Results show that current open-source SDMs can handle everyday spoken conversations, but still lag behind cascaded or proprietary systems in instruction following and reasoning. At the same time, the pro-track results indicate remaining weaknesses in speech-specific interaction abilities, including speaker awareness, cross-lingual oral interaction, and acoustically grounded understanding. These findings suggest that future S2S systems should be evaluated by both cognitive capability retention and speech-specific capability acquisition.
- Open Access
- 12 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 20 (1) , 63-76
SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.
- 12 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 20 (1) , 63-76
Open AccessSLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 10748-10757
EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 10748-10757
Open AccessEmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting