View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (10) , 10076-10084
Abstract
View transformation robustness (VTR) is critical for deep-learning-based multi-view 3D object reconstruction models, which indicates the methods' stability under inputs with various view transformations. However, existing research seldom focused on view transformation robustness in multi-view 3D object reconstruction. One direct way to improve the models' VTR is to produce data with more view transformations and add them to model training. Recent progress on large vision models, particularly Stable Diffusion models, has provided great potential for generating 3D models or synthesizing novel view images with only a single image input. Directly deploying these models at inference consumes heavy computation resources and their robustness to view transformations is not guaranteed either. To fully utilize the power of Stable Diffusion models without extra inference computation burdens, we propose to generate novel views with Stable Diffusion models for better view transformation robustness. Instead of synthesizing random views, we propose a reconstruction error-guided view selection method, which considers the reconstruction errors' spatial distribution of the 3D predictions and chooses the views that could cover the reconstruction errors as much as possible. The methods are trained and tested on sets with large view transformations to validate the 3D reconstruction models' robustness to view transformations. Extensive experiments demonstrate that the proposed method can outperform state-of-the-art 3D reconstruction methods and other view transformation robustness comparison methods.Keywords
This publication has 10 references indexed in Scilit:
- MVTN: Learning Multi-view Transformations for 3D UnderstandingInternational Journal of Computer Vision, 2024
- Challenges for Monocular 6-D Object Pose Estimation in RoboticsIEEE Transactions on Robotics, 2024
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion2024
- Neural Wavelet-domain Diffusion for 3D Shape Generation, Inversion, and ManipulationACM Transactions on Graphics, 2024
- GARNet: Global-aware multi-view 3D reconstruction network and the cost-performance tradeoffPattern Recognition, 2023
- R₂FD₂: Fast and Robust Matching of Multimodal Remote Sensing Images via Repeatable Feature Detector and Rotation-Invariant Feature DescriptorIEEE Transactions on Geoscience and Remote Sensing, 2023
- DreamSparse: Escaping from Plato’s Cave with 2D Diffusion Model Given Sparse Views2023
- 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers2021
- Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple ImagesInternational Journal of Computer Vision, 2020
- V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation2016