DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis
Preprint
Abstract
While recent video-to-audio (V2A) models can generate realistic background audio from visual input, they largely overlook speech, an essential part of many video soundtracks. This paper proposes a new task, video-to-soundtrack (V2ST) generation, which aims to jointly produce synchronized background audio and speech within a unified framework. To tackle V2ST, we introduce DualDub, a unified framework built on a multimodal language model that integrates a multimodal encoder, a cross-modal aligner, and dual decoding heads for simultaneous background audio and speech generation. Specifically, our proposed cross-modal aligner employs causal and non-causal attention mechanisms to improve synchronization and acoustic harmony. Besides, to handle data scarcity, we design a curriculum learning strategy that progressively builds the multimodal capability. Finally, we introduce DualBench, the first benchmark for V2ST evaluation with a carefully curated test set and comprehensive metrics. Experimental results demonstrate that DualDub achieves state-of-the-art performance, generating high-quality and well-synchronized soundtracks with both speech and background audio.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- KALL-E: Autoregressive Speech Synthesis with Next-Distribution PredictionProceedings of the AAAI Conference on Artificial Intelligence, 2026
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized SoundsInternational Journal of Computer Vision, 2026
- FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching2025
- V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow2025
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching2025
- Autoregressive Speech Synthesis without Vector Quantization2025
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model AdaptationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 20222022
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019