CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigation
- 1 November 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (11) , 14239-14253
Abstract
Embodied navigation is a core task in embodied AI. It requires comprehensive scene understanding and precise spatial reasoning. Recent vision-language models (VLMs) with strong generalization capabilities and rich commonsense knowledge have shown remarkable performance when applied to embodied navigation tasks. However, these models still encounter insufficient understanding of 3D geometry and spatial semantics when applied to real-world 3D navigation. To address this, we propose CoNav, a collaborative cross-modal reasoning framework. First, we pretrain a 3D-language model with the curriculum learning schedule and prepare a pretrained vision-language navigation agent. Next, with lightweight fine-tuning on a small 2D-3D-text corpus, the vision-language navigation agent learns to combine visual evidence with knowledge from the 3D-language model. Finally, the pretrained 3D-language model communicates with the vision-language navigation agent, enabling collaborative cross-modal reasoning and resolving ambiguities during navigation. This yields more reliable and efficient image-3D fusion for embodied navigation. CoNav introduces a new collaborative framework between a 3D-language model and a vision-language navigation agent for embodied navigation. Notably, CoNav requires only a small 2D-3D-text corpus to align 3D and 2D data. CoNav achieves clear improvements on four standard embodied navigation benchmarks (R2R, CVDN, REVERIE, SOON) and two spatial reasoning benchmarks (ScanQA, SQA3D). Moreover, Under similar success rates, it also finds shorter paths than prior methods, as measured by SPL. The results demonstrate the value of collaborative 2D-3D reasoning for embodied navigation.Keywords
All Related Versions
Funding Information
- New Generation Artificial Intelligence-National Science and Technology Major Projection (2025ZD0123100)
- National Natural Science Foundation of China (62573399)
This publication has 60 references indexed in Scilit:
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion ModelsInternational Journal of Computer Vision, 2024
- PointLLM: Empowering Large Language Models to Understand Point Clouds2024
- SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding2024
- NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models2024
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language2020
- Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation2019
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments2018
- Semantic Scene Completion from a Single Depth Image2017
- ImageNet: A large-scale hierarchical image database2009