NeRF-MAE : Masked AutoEncoders for Self Supervised 3D representation Learning for Neural Radiance Fields
Authors:
- Muhammad Zubair IrshadMIMuhammad Zubair Irshad
- Sergey ZakahrovSZSergey Zakahrov
- Vitor GuiziliniVGVitor Guizilini
- Adrien GaidonAGAdrien Gaidon
- Zsolt KiraZKZsolt Kira
- Rares AmbrusRARares Ambrus
Preprint
Abstract
Neural fields excel in computer vision and robotics due to their ability to understand the 3D visual world such as inferring semantics, geometry, and dynamics. Given the capabilities of neural fields in densely representing a 3D scene from 2D images, we ask the question: Can we scale their self-supervised pretraining, specifically using masked autoencoders, to generate effective 3D representations from posed RGB images. Owing to the astounding success of extending transformers to novel data modalities, we employ standard 3D Vision Transformers to suit the unique formulation of NeRFs. We leverage NeRF's volumetric grid as a dense input to the transformer, contrasting it with other 3D representations such as pointclouds where the information density can be uneven, and the representation is irregular. Due to the difficulty of applying masked autoencoders to an implicit representation, such as NeRF, we opt for extracting an explicit representation that canonicalizes scenes across domains by employing the camera trajectory for sampling. Our goal is made possible by masking random patches from NeRF's radiance and density grid and employing a standard 3D Swin Transformer to reconstruct the masked patches. In doing so, the model can learn the semantic and spatial structure of complete scenes. We pretrain this representation at scale on our proposed curated posed-RGB data, totaling over 1.6 million images. Once pretrained, the encoder is used for effective 3D transfer learning. Our novel self-supervised pretraining for NeRFs, NeRF-MAE, scales remarkably well and improves performance on various challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining, NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF scene understanding baselines on Front3D and ScanNet datasets with an absolute performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive ReviewInternational Journal of Computer Vision, 2026
- Gaussian Grouping: Segment and Edit Anything in 3D Scenes2024
- STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Rendering2021
- Holistic 3D Scene Understanding from a Single Image with Implicit Representation2021
- D-NeRF: Neural Radiance Fields for Dynamic Scenes2021
- Neural Scene Graphs for Dynamic Scenes2021
- Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes2021
- pixelNeRF: Neural Radiance Fields from One or Few Images2021
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision2020