Probing the 3D Awareness of Visual Foundation Models
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 21795-21806
Abstract
Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and local- ize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen fea- tures. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- DeiT III: Revenge of the ViT2022
- Zero-Shot Category-Level Object Pose Estimation2022
- Discriminatively Trained Dense Surface Normal Estimation2014
- Indoor Segmentation and Support Inference from RGBD Images2012
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Surface shape and curvature scalesImage and Vision Computing, 1992
- Symbolic reasoning among 3-D models and 2-D imagesArtificial Intelligence, 1981
- The internal representation of solid shape with respect to visionBiological Cybernetics, 1979
- The singularities of the visual mappingBiological Cybernetics, 1976
- Second-order isomorphism of internal representations: Shapes of statesCognitive Psychology, 1970