Continuous 3D Perception Model with Persistent State
Authors:
Top Cited Papers
- 10 June 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10510-10522
Abstract
We present a unified framework capable of solving a broad range of 3D tasks. Our approach features a stateful recurrent model that continuously updates its state representation with each new observation. Given a stream of images, this evolving state can be used to generate metric-scale pointmaps (per-pixel 3D points) for each new input in an online fashion. These pointmaps reside within a common coordinate system, and can be accumulated into a coherent, dense scene reconstruction that updates as new images arrive. Our model, called CUT3R (Continuous Updating Transformer for 3D Reconstruction), captures rich priors of real-world scenes: not only can it predict accurate pointmaps from image observations, but it can also infer unseen regions of the scene by probing at virtual, unobserved views. Our method is simple yet highly flexible, naturally accepting varying length of images that may be either video streams or unordered photo collections, containing both static and dynamic content. We evaluate our method on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each. Project page: https://cut3r.github.io/.Keywords
All Related Versions
This publication has 78 references indexed in Scilit:
- Grounding Image Matching in 3D with MASt3R2024
- TensoRF: Tensorial Radiance Fields2022
- ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild2022
- Structure and Motion from Casual Videos2022
- SimpleRecon: 3D Reconstruction Without 3D Convolutions2022
- Map-Free Visual Relocalization: Metric Pose Relative to a Single Image2022
- MVSNet: Depth Inference for Unstructured Multi-view Stereo2018
- 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction2016
- LSD-SLAM: Large-Scale Direct Monocular SLAM2014
- Bundle Adjustment — A Modern Synthesis2000