Multisource Region Attention Network for Fine-Grained Object Recognition in Remote Sensing Imagery
Preprint
Abstract
Fine-grained object recognition concerns the identification of the type of an object among a large number of closely related sub-categories. Multisource data analysis, that aims to leverage the complementary spectral, spatial, and structural information embedded in different sources, is a promising direction towards solving the fine-grained recognition problem that involves low between-class variance, small training set sizes for rare classes, and class imbalance. However, the common assumption of co-registered sources may not hold at the pixel level for small objects of interest. We present a novel methodology that aims to simultaneously learn the alignment of multisource data and the classification model in a unified framework. The proposed method involves a multisource region attention network that computes per-source feature representations, assigns attention scores to candidate regions sampled around the expected object locations by using these representations, and classifies the objects by using an attention-driven multisource representation that combines the feature representations and the attention scores from all sources. All components of the model are realized using deep neural networks and are learned in an end-to-end fashion. Experiments using RGB, multispectral, and LiDAR elevation data for classification of street trees showed that our approach achieved 64.2% and 47.3% accuracies for the 18-class and 40-class settings, respectively, which correspond to 13% and 14.3% improvement relative to the commonly used feature concatenation approach from multiple sources.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- From Google Maps to a fine-grained catalog of street treesISPRS Journal of Photogrammetry and Remote Sensing, 2018
- Mapping urban tree species using integrated airborne hyperspectral and LiDAR remote sensing dataRemote Sensing of Environment, 2017
- Review of studies on tree species classification from remotely sensed dataRemote Sensing of Environment, 2016
- Processing of Extremely High-Resolution LiDAR and RGB Data: Outcome of the 2015 IEEE GRSS Data Fusion Contest–Part A: 2-D ContestIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2016
- Processing of Multiresolution Thermal Hyperspectral and Digital Color Data: Outcome of the 2014 IEEE GRSS Data Fusion ContestIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015
- Ensemble Multiple Kernel Active Learning For Classification of Multisource Remote Sensing DataIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014
- Urban tree species mapping using hyperspectral and lidar data fusionRemote Sensing of Environment, 2014
- Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 GRSS Data Fusion ContestIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014
- Tree species classification in the Southern Alps based on the fusion of very high geometrical resolution multispectral/hyperspectral images and LiDAR dataRemote Sensing of Environment, 2012
- Classification of savanna tree species, in the Greater Kruger National Park region, by integrating hyperspectral and LiDAR data in a Random Forest data mining environmentISPRS Journal of Photogrammetry and Remote Sensing, 2012