OCC-MLLM-CoT-Alpha: Towards Multi-stage Occlusion Recognition Based on Large Language Models via 3D-Aware Supervision and Chain-of-Thoughts Guidance
Preprint
Abstract
Comprehending occluded objects are not well studied in existing large-scale visual-language multi-modal models. Current state-of-the-art multi-modal large models struggles to provide satisfactory results in understanding occluded objects through universal visual encoders and supervised learning strategies. Therefore, we propose OCC-MLLM-CoT-Alpha, a multi-modal large vision language framework that integrates 3D-aware supervision and Chain-of-Thoughts guidance. Particularly, (1) we build a multi-modal large vision-language model framework which is consisted of a large multi-modal vision-language model and a 3D reconstruction expert model. (2) the corresponding multi-modal Chain-of-Thoughts is learned through a combination of supervised and reinforcement training strategies, allowing the multi-modal vision-language model to enhance the recognition ability with learned multi-modal chain-of-thoughts guidance. (3) A large-scale multi-modal chain-of-thoughts reasoning dataset, consisting of $110k$ samples of occluded objects held in hand, is built. In the evaluation, the proposed methods demonstrate decision score improvement of 15.75%,15.30%,16.98%,14.62%, and 4.42%,3.63%,6.94%,10.70% for two settings of a variety of state-of-the-art models.
Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Binary Classifier Optimization for Large Language Model Alignment2025
- MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversityScience China Information Sciences, 2024
- MOHO: Learning Single-View Hand-Held Object Reconstruction with Multi-View Occlusion-Aware Supervision2024
- An Advanced Approach to Object Detection and Tracking in Robotics and Autonomous Vehicles Using YOLOv8 and LiDAR Data FusionElectronics, 2024
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs2024
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Learning Joint Reconstruction of Hands and Manipulated Objects2019
- Hand-Object Contact Force Estimation from Markerless Visual TrackingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016