DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos
Preprint
Abstract
Currently, high-quality, synchronized audio is synthesized from video and optional text inputs using various multi-modal joint learning frameworks. However, the precise alignment between the visual and generated audio domains remains far from satisfactory. One key factor is the lack of sufficient temporal and semantic alignment annotations in open-source video-audio and text-audio benchmarks. Therefore, we propose a framework for audio generation from videos, leveraging the internal chain-of-thought (CoT) of a multi-modal large language model (MLLM) to enable step-by-step reasoning without requiring additional annotations. Additionally, a corresponding multi-modal reasoning dataset is constructed to facilitate the learning of initial reasoning in audio generation. In the experiments, we demonstrate the effectiveness of the proposed framework in reducing misalignment (voice-over) in generated audio and achieving competitive performance compared to various state-of-the-art models. The evaluation results show that the proposed method outperforms state-of-the-art approaches across multiple metrics. Specifically, the F DP aSST indicator is reduced by up to 10.07%, the F DP AN N s indicator by up to 11.62%, and the F DV GG indicator by up to 38.61%. Furthermore, the IS indicator improves by up to 4.95%, the IB-score indicator increases by up to 6.39%, and the DeSync indicator is reduced by up to 0.89%.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized SoundsInternational Journal of Computer Vision, 2026
- Video-Foley: Two-Stage Video-to-Sound Generation via Temporal Event Condition for Foley SoundIEEE Transactions on Audio, Speech and Language Processing, 2025
- TokenPacker: Efficient Visual Projector for Multimodal LLMInternational Journal of Computer Vision, 2025
- Video-Guided Foley Sound Generation with Multimodal Controls2025
- Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS ControlProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis2025
- Temporally Aligned Audio for Video with Autoregression2025
- A review of emerging research directions in Abstract Visual ReasoningInformation Fusion, 2022
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- Generating Visually Aligned Sound From VideosIEEE Transactions on Image Processing, 2020