Classification of Hyperspectral and LiDAR Data Using Multi-Modal Transformer Cascaded Fusion Net
Open Access
- 24 August 2023
- journal article
- research article
- Published by MDPI AG in Remote Sensing
- Vol. 15 (17) , 4142
Abstract
With the continuous development of surface observation methods and technologies, we can acquire multiple sources of data more effectively in the same geographic area. The quality and availability of these data have also significantly improved. Consequently, how to better utilize multi-source data to represent ground information has become an important research question in the field of geoscience. In this paper, a novel model called multi-modal transformer cascaded fusion net (MMTCFN) is proposed for fusion and classification of multi-modal remote sensing data, Hyperspectral Imagery (HSI) and LiDAR data. Feature fusion and feature extraction are the two stages of the model. First, in the feature extraction stage, a three-branch cascaded Convolutional Neural Network (CNN) framework is employed to fully leverage the advantages of convolutional operators in extracting shallow-level local features. Based on this, we generated multi-modal long-range integrated deep features utilizing the transformer-based vectorized pixel group transformer (VPGT) module during the feature fusion stage. In the VPGT block, we designed a vectorized pixel group embedding that preserves the global features extracted from the three branches in a non-overlapping multi-space manner. Moreover, we introduce the DropKey mechanism into the multi-head self-attention (MHSA) to alleviate overfitting caused by insufficient training samples. Finally, we employ a probabilistic decision fusion strategy to integrate multiple class estimations, assigning a specific category to each pixel. This model was experimented on three HSI-LiDAR datasets with balanced and unbalanced training samples. The proposed model outperforms the other seven SOTA approaches in terms of OA performance, proving the superiority of MMTCFN for the HSI-LiDAR classification task.Keywords
Funding Information
- National Key Research and Development Program of China (2018YFB0504504, AR2201, AR2203)
- Funded Project of Fundamental Scientific Research Business Expenses of Chinese Academy of Surveying and Mapping (2018YFB0504504, AR2201, AR2203)
This publication has 48 references indexed in Scilit:
- 3-D Deep Learning Approach for Remote Sensing Image ClassificationIEEE Transactions on Geoscience and Remote Sensing, 2018
- Hyperspectral Image Classification With Deep Feature Fusion NetworkIEEE Transactions on Geoscience and Remote Sensing, 2018
- Deep Learning in Remote Sensing: A Comprehensive Review and List of ResourcesIEEE Geoscience and Remote Sensing Magazine, 2017
- Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component AnalysisIEEE Transactions on Geoscience and Remote Sensing, 2017
- Mining Coastal Land Use Sequential Pattern and Its Land Use Associations Based on Association Rule MiningRemote Sensing, 2017
- Deep Residual Learning for Image Recognition2016
- Optimal vegetation cover in the Horqin Sands, ChinaEcohydrology, 2015
- Fusion of hyperspectral and LIDAR data using decision template-based fuzzy multiple classifier systemInternational Journal of Applied Earth Observation and Geoinformation, 2015
- UL-Isomap based nonlinear dimensionality reduction for hyperspectral imagery classificationISPRS Journal of Photogrammetry and Remote Sensing, 2014
- Locality-Preserving Discriminant Analysis in Kernel-Induced Feature Spaces for Hyperspectral Image ClassificationIEEE Geoscience and Remote Sensing Letters, 2011