SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video
- 11 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-15
Abstract
Visual-spatial understanding, defined as the ability to infer object relationships and scene layouts from visual inputs, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, pre-trained vision-language models (VLMs) remain constrained by spatial uncertainty stemming from inherently 2D observations and by the scarcity of data for 3D spatial understanding. To address these limitations, we proposed a novel framework, SpaceEra, in the NeurIPS 2025 Spotlight paper. Although it achieved significant performance gains, we further observed that its effectiveness is hindered by insufficient input from scanning videos and weak reasoning constraints. To tackle these newly emerged challenges, we extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference. Specifically, to alleviate input insufficiency, we introduce ScenePick, a frame sampling strategy that balances spatial coverage with object semantics to produce compact yet comprehensive scene representations. In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations, thereby aligning optimization with spatial accuracy. Extensive experiments across multiple benchmarks demonstrate consistent improvements over strong baselines, while ablation studies validate both the individual and joint contributions of each component, and further analyses provide guidance for future research.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U24A20328)
- Shenzhen Science and Technology Program (KQTD20240729102207002)
- Guangdong S&T Program (2025B0101130003)
- Guangdong Basic and Applied Basic Research Foundation (2025A1515011732)
- Beijing Natural Science Foundation (4262074)
- Science and Technology Innovation Program for Distinguished Young Scholars of Shandong Province (2023KJ128)
- Major Key Project of Pengcheng Laboratory (PCL2025A14-4)
- Fundamental Research Funds for the Central Universities (HIT.DZJJ.2025048)
This publication has 39 references indexed in Scilit:
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningNature, 2025
- PointLLM: Empowering Large Language Models to Understand Point Clouds2024
- ShapeLLM: Universal 3D Object Understanding for Embodied Interaction2024
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance2024
- Point Transformer V3: Simpler, Faster, Stronger2024
- Objaverse: A Universe of Annotated 3D Objects2023
- SoftGroup for 3D Instance Segmentation on Point Clouds2022
- ScanQA: 3D Question Answering for Spatial Scene Understanding2022
- Matterport3D: Learning from RGB-D Data in Indoor Environments2017
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes2017