When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
Preprint
Abstract
Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an ``Audio-Visual Confusion'' scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs Is there a/an muted-object sound''. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves the accuracy by 10~30\% over the baseline model with limited training data. Follow: https://github.com/rikeilong/AVConfusion.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- CAT+: Investigating and Enhancing Audio-visual Understanding in Large Language ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Reliable and Balanced Transfer Learning for Generalized Multimodal Face Anti-SpoofingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- FusionMamba: dynamic feature enhancement for multimodal image fusion with MambaVisual Intelligence, 2024
- Pose-Promote: Progressive Visual Perception for Activities of Daily LivingIEEE Signal Processing Letters, 2024
- Detecting hallucinations in large language models using semantic entropyNature, 2024
- OneLLM: One Framework to Align All Modalities with Language2024
- 3sG: Three‐stage guidance for indoor human action recognitionIET Image Processing, 2024
- CAD - Contextual Multi-modal Alignment for Dynamic AVQA2024
- Learning to Answer Questions in Dynamic Audio-Visual Scenarios2022
- Pano-AVQA: Grounded Audio-Visual Question Answering on 360° Videos2021