DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model With Self-Generated Cross-Modal Alignment
Top Cited Papers
- 19 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 2062-2076
Abstract
We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Language Models (LLMs) with auditory capabilities by training on large-scale audio-instruction datasets. However, existing LALMs have often suffered from the catastrophic forgetting of the LLM's original abilities. Therefore, balancing knowledge retention and audio perception has become a critical challenge. To address this, we revisit the data construction pipeline and propose a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets, named DeSTA. This approach aims at preserving the LLM's native language proficiency thereby enabling zero-shot generalization without task-specific tuning. We construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms existing training strategies. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs.Keywords
All Related Versions
Funding Information
- National Science and Technology Council (114-2628-E-002-022)
- NVIDIA academic
This publication has 73 references indexed in Scilit:
- Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model2025
- AI models collapse when trained on recursively generated dataNature, 2024
- VILA: On Pre-training for Visual Language Models2024
- Joint Audio and Speech Understanding2023
- An Intelligent Infrastructure Toward Large Scale Naturalistic Affective Speech Corpora Collection2023
- Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset2021
- Clotho: an Audio Captioning Dataset2020
- Audio Set: An ontology and human-labeled dataset for audio events2017
- Modal analysis and transcription of strokes of the mridangam using non-negative matrix factorization2013
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008