SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
Open Access
- 12 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 20 (1) , 63-76
Abstract
The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U23B2018)
- Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102)
- Yangtze River Delta Science and Technology Innovation Community Joint Research (2024CSJGG01100)
This publication has 55 references indexed in Scilit:
- AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face GenerationIEEE Access, 2024
- EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning2024
- Extending Large Language Models for Speech and Audio Captioning2024
- Can Audio Captions Be Evaluated With Image Caption Metrics?2022
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020
- Clotho: an Audio Captioning Dataset2020
- Audio Set: An ontology and human-labeled dataset for audio events2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- CIDEr: Consensus-based image description evaluation2015
- Librispeech: An ASR corpus based on public domain audio books2015