Deep Network Architectures as Feature Extractors for Multi-Label Classification of Remote Sensing Images
Open Access
- 16 January 2023
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 15 (2) , 538
Abstract
Data in the form of images are now generated at an unprecedented rate. A case in point is remote sensing images (RSI), now available in large-scale RSI archives, which have attracted a considerable amount of research on image classification within the remote sensing community. The basic task of single-target multi-class image classification considers the case where each image is assigned exactly one label from a predefined finite set of class labels. Recently, however, image annotations have become increasingly complex, with images labeled with several labels (instead of just one). In other words, the goal is to assign multiple semantic categories to an image, based on its high-level context. The corresponding machine learning tasks is called multi-label classification (MLC). The classification of RSI is currently predominantly addressed by deep neural network (DNN) approaches, especially convolutional neural networks (CNNs), which can be utilized as feature extractors as well as end-to-end methods. After only considering single-target classification for a long period, DNNs have recently emerged that address the task of MLC. On the other hand, trees and tree ensembles for MLC have a long tradition and are the best-performing class of MLC methods, but need predefined feature representations to operate on. In this work, we explore different strategies for model training based on the transfer learning paradigm, where we utilize different families of (pre-trained) CNN architectures, such as VGG, EfficientNet, and ResNet. The architectures are trained in an end-to-end manner and used in two different modes of operation, namely, as standalone models that directly perform the MLC task, and as feature extractors. In the latter case, the learned representations are used with tree ensemble methods for MLC, such as random forests and extremely randomized trees. We conduct an extensive experimental analysis of methods over several publicly available RSI datasets and evaluate their effectiveness in terms of standard MLC measures. Of these, ranking-based evaluation measures are most relevant, especially ranking loss. The results show that, for addressing the RSI-MLC task, it is favorable to use lightweight network architectures, such as EfficientNet-B2, which is the best performing end-to-end approach, as well as a feature extractor. Furthermore, in the datasets with a limited number of images, using traditional tree ensembles for MLC can yield better performance compared to end-to-end deep approaches.Keywords
Funding Information
- European Space Agency (P2-0103, J2-2505)
- The Slovenian Research and Innovation Agency (P2-0103, J2-2505)
This publication has 31 references indexed in Scilit:
- Convolutional Neural Network for Remote-Sensing Scene Classification: Transfer Learning AnalysisRemote Sensing, 2019
- Recurrently exploring class-wise attention in a hybrid convolutional and bidirectional LSTM network for multi-label aerial image classificationISPRS Journal of Photogrammetry and Remote Sensing, 2019
- Deep Transfer Learning for Crop Yield Prediction with Remote Sensing Data2018
- Multilabel Remote Sensing Image Retrieval Using a Semisupervised Graph-Theoretic MethodIEEE Transactions on Geoscience and Remote Sensing, 2017
- AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene ClassificationIEEE Transactions on Geoscience and Remote Sensing, 2017
- Content based hyperspectral image retrieval using bag of endmembers image descriptors2016
- CNN-RNN: A Unified Framework for Multi-label Image Classification2016
- Tree ensembles for predicting structured outputsPattern Recognition, 2013
- Bag-of-visual-words and spatial extensions for land-use classification2010
- ImageNet: A large-scale hierarchical image database2009