Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning
Authors:
- 16 July 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4966-4969
Abstract
The development of federated learning (FL) methods, which aim to learn from distributed databases (i.e., clients) without accessing data on clients, has recently attracted great attention. Most of these methods assume that the clients are associated with the same data modality. However, remote sensing (RS) images in different clients can be associated with different data modalities that can improve the classification performance when jointly used. To address this problem, in this paper we introduce a novel multi-modal FL framework that aims to learn from decentralized multi-modal RS image archives for RS image classification problems. The proposed framework is made up of three modules: 1) multimodal fusion (MF); 2) feature whitening (FW); and 3) mutual information maximization (MIM). The MF module performs iterative model averaging to learn without accessing data on clients in the case that clients are associated with different data modalities. The FW module aligns the representations learned among the different clients. The MIM module maximizes the similarity of images from different modalities. Experimental results show the effectiveness of the proposed framework compared to iterative model averaging, which is a widely used algorithm in FL. The code of the proposed framework is publicly available at https://git.tu-berlin.de/rsim/MMFL.Keywords
All Related Versions
Funding Information
- European Research Council
This publication has 11 references indexed in Scilit:
- Prototype-Based Clustered Federated Learning for Semantic Segmentation of Aerial Images2022
- FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction2022
- Layer-wised Model Aggregation for Personalized Federated Learning2022
- Progress and Challenges in Intelligent Remote Sensing Satellite SystemsIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022
- BigEarthNet-MM: A Large-Scale, Multimodal, Multilabel Benchmark Archive for Remote Sensing Image Classification and Retrieval [Software and Data Sets]IEEE Geoscience and Remote Sensing Magazine, 2021
- Model-Contrastive Federated Learning2021
- L(1/2) Regularization for ISAR Imaging and Target Enhancement of Complex ImageIEEE Transactions on Geoscience and Remote Sensing, 2020
- More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery ClassificationIEEE Transactions on Geoscience and Remote Sensing, 2020
- Unsupervised Domain Adaptation Using Feature-Whitening and Consensus Loss2019
- Deep Residual Learning for Image Recognition2016