DeepDubber-V1: Towards High Quality and Dialogue, Narration, Monologue Adaptive Movie Dubbing Via Multi-Modal Chain-of-Thoughts Reasoning Guidance
Preprint
Abstract
Current movie dubbing technology can generate the desired voice from a given speech prompt, ensuring good synchronization between speech and visuals while accurately conveying the intended emotions. However, in movie dubbing, key aspects such as adapting to different dubbing styles, handling dialogue, narration, and monologue effectively, and understanding subtle details like the age and gender of speakers, have not been well studied. To address this challenge, we propose a framework of multi-modal large language model. First, it utilizes multimodal Chain-of-Thought (CoT) reasoning methods on visual inputs to understand dubbing styles and fine-grained attributes. Second, it generates high-quality dubbing through large speech generation models, guided by multimodal conditions. Additionally, we have developed a movie dubbing dataset with CoT annotations. The evaluation results demonstrate a performance improvement over state-of-the-art methods across multiple datasets. In particular, for the evaluation metrics, the SPK-SIM and EMO-SIM increases from 82.48% to 89.74%, 66.24% to 78.88% for dubbing setting 2.0 on V2C Animation dataset, LSE-D and MCD-SL decreases from 14.79 to 14.63, 5.24 to 4.74 for dubbing setting 2.0 on Grid dataset, SPK-SIM increases from 64.03 to 83.42 and WER decreases from 52.69% to 23.20% for initial reasoning setting on proposed CoT-Movie-Dubbing dataset in the comparison with the state-of-the art models.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding2025
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models2025
- Audiopedia: Audio QA with Knowledge2025
- Emilia: An Extensive, Multilingual, and Diverse Speech Dataset For Large-Scale Speech Generation2024
- From image to language: A critical analysis of Visual Question Answering (VQA) approaches, challenges, and opportunitiesInformation Fusion, 2024
- Imaginary Voice: Face-Styled Diffusion Model for Text-to-Speech2023
- Learning to Dub Movies via Hierarchical Prosody Models2023
- A review of emerging research directions in Abstract Visual ReasoningInformation Fusion, 2022
- V2C: Visual Voice Cloning2022
- Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis2020