FreeOrbit4D: Training-Free Arbitrary Camera Redirection for Monocular Videos via Foreground-Complete 4D Reconstruction
Top Cited Papers
Open Access
- 19 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Abstract
Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/Keywords
Funding Information
- National Artificial Intelligence Research Resource (NAIRR) Pilot (NAIRR250199)
- Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (CIS250012)
- Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (CIS250816)
- Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (CIS251188)
- Nvidia (Academic Grant Program)
- National Science Foundation (#2525287)
- National Science Foundation (#2404385)
- National Science Foundation (#2414227)
- National Science Foundation (#2340254)
- National Science Foundation (#2312102)
- National Science Foundation (#2331878)
This publication has 39 references indexed in Scilit:
- ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning2025
- MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds2025
- Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video2025
- Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control2025
- VGGT: Visual Geometry Grounded Transformer2025
- Predicting the Road Ahead: A Knowledge Graph Based Foundation Model for Scene Understanding in Autonomous Driving2025
- PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction2024
- Motion2VecSets: 4D Latent Vector Set Diffusion for Non-Rigid Shape Reconstruction and Tracking2024
- VBench: Comprehensive Benchmark Suite for Video Generative Models2024
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016