Remote Sensing Neural Radiance Fields for Multi-View Satellite Photogrammetry
Open Access
- 31 July 2023
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 15 (15) , 3808
Abstract
Neural radiance fields (NeRFs) combining machine learning with differentiable rendering have arisen as one of the most promising approaches for novel view synthesis and depth estimates. However, NeRFs only applies to close-range static imagery and it takes several hours to train the model. The satellites are hundreds of kilometers from the earth. Satellite multi-view images are usually captured over several years, and the scene of images is dynamic in the wild. Therefore, multi-view satellite photogrammetry is far beyond the capabilities of NeRFs. In this paper, we present a new method for multi-view satellite photogrammetry of Earth observation called remote sensing neural radiance fields (RS-NeRFs). It aims to generate novel view images and accurate elevation predictions quickly. For each scene, we train an RS-NeRF using high-resolution optical images without labels or geometric priors and apply image reconstruction losses for self-supervised learning. Multi-date images exhibit significant changes in appearance, mainly due to cars and varying shadows, which brings challenges to satellite photogrammetry. Robustness to these changes is achieved by the input of solar ray direction and the vehicle removal method. NeRFs make it intolerable by requiring a very long time to train an easy scene. In order to significantly reduce the training time of RS-NeRFs, we build a tiny network with HashEncoder and adopted a new sampling technique with our custom CUDA kernels. Compared with previous work, our method performs better on novel view synthesis and elevation estimates, taking several minutes.Keywords
Funding Information
- Science and Technology Plan Transfer Payment (2021ZYSF007, 2020CDYB-29, 2021KJT0012-2021YFS0067)
- Sichuan University and Yibin Municipal People’s Government University (2021ZYSF007, 2020CDYB-29, 2021KJT0012-2021YFS0067)
- Key Research and Development Program of Science and Technology Department of Sichuan Province (2021ZYSF007, 2020CDYB-29, 2021KJT0012-2021YFS0067)
This publication has 45 references indexed in Scilit:
- 3D change detection – Approaches and applicationsISPRS Journal of Photogrammetry and Remote Sensing, 2016
- The High Resolution Stereo Camera (HRSC) of Mars Express and its approach to science analysis and mapping for Mars and its satellitesPlanetary and Space Science, 2016
- An automated, open-source pipeline for mass production of digital elevation models (DEMs) from very-high-resolution commercial stereo satellite imageryISPRS Journal of Photogrammetry and Remote Sensing, 2016
- Building Rome in a dayCommunications of the ACM, 2011
- Modeling and Recognition of Landmark Image Collections Using Iconic Scene Graphs2008
- Stereo Processing by Semiglobal Matching and Mutual InformationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2007
- Detailed Real-Time Urban 3D Reconstruction from VideoInternational Journal of Computer Vision, 2007
- Mapping Height and Biomass of Mangrove Forests in Everglades National Park with SRTM Elevation DataPhotogrammetric Engineering & Remote Sensing, 2006
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- An Automated Method for Large-Scale, Ground-Based City Model AcquisitionInternational Journal of Computer Vision, 2004