One for All: Toward Unified Foundation Models for Earth Vision
Authors:
- 7 July 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2734-2738
Abstract
Foundation models characterized by extensive parameters and trained on large-scale datasets have demonstrated remarkable efficacy across various downstream tasks for remote sensing data. Current remote sensing foundation models typically specialize in a single modality or a specific spatial resolution range, limiting their versatility for downstream datasets. While there have been attempts to develop multi-modal remote sensing foundation models, they typically employ separate vision encoders for each modality or spatial resolution, necessitating a switch in backbones contingent upon the input data. To address this issue, we introduce a simple yet effective method, termed OFA-Net (One-For-All Network): employing a single, shared Transformer backbone for multiple data modalities with different spatial resolutions. Using the masked image modeling mechanism, we pre-train a single Transformer backbone on a curated multi-modal dataset with this simple design. Then the backbone model can be used in different downstream tasks, thus forging a path towards a unified foundation backbone model in Earth vision. The proposed method is evaluated on 12 distinct downstream tasks and demonstrates promising performance.Keywords
All Related Versions
Funding Information
- Ministry of Education
- Nature
This publication has 8 references indexed in Scilit:
- A Billion-Scale Foundation Model for Remote Sensing ImagesIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024
- SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding2023
- SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation [Software and Data Sets]IEEE Geoscience and Remote Sensing Magazine, 2023
- THE Benchmark: Transferable Representation Learning for Monocular Height EstimationIEEE Transactions on Geoscience and Remote Sensing, 2023
- HySpecNet-11k: a Large-Scale Hyperspectral Dataset for Benchmarking Learning-Based Hyperspectral Image Compression Methods2023
- Enabling country-scale land cover mapping with meter-resolution satellite imageryISPRS Journal of Photogrammetry and Remote Sensing, 2023
- Self-Supervised Learning for Invariant Representations From Multi-Spectral and SAR ImagesIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022
- Deep Learning in Remote Sensing: A Comprehensive Review and List of ResourcesIEEE Geoscience and Remote Sensing Magazine, 2017