Bilinear Parallel Fourier Transformer for Multimodal Remote Sensing Classification

Authors:
Publisher website
Google Scholar
Abstract
Vision transformers (ViTs) have shown promise in multimodal fusion image classification, yet face performance challenges in complex remote sensing scenarios. Single fusion frameworks often fail to fully utilize multimodal diversity, and the uneven distribution of image categories complicates the accurate construction of spatial structures by transformers. Additionally, traditional cross-entropy tends to favor majority classes, neglecting minority classes, resulting in suboptimal predictions and reduced overall accuracy. To solve these challenges, we propose a novel deep neural network, bilinear parallel Fourier transformer (BPFT). We propose a novel dual fusion feature interaction (DFFI) module which utilize two distinct types of fused features for learning, namely spatial-spectral fusion feature and global fusion feature. Besides, we introduce a dual feature interaction module to improve the utilization of fused feature information. To enable the transformer to better establish spatial structural relationships, we employ the Fourier transform in place of the self-attention mechanism. To address the focus on minority class labels, we propose an exponential label smoothing cross-entropy loss function. This loss function comprises two components: exponential cross-entropy and label smoothing. The exponential cross-entropy component applies a strong penalty to misclassified samples, thereby increasing attention on minority class labels. To validate the efficacy of our approach, extensive experiments are conducted across two multimodal remote sensing datasets: Augsburg and Berlin, encompassing HSI data and SAR data. The results of these experiments affirm the superior performance of our proposed BPFT model compared to existing state-of-the-art models in multimodal remote sensing image classification tasks.
Funding Information
  • Hainan Province “Nanhai New Star” Technology Innovation Talent Platform Project (NHXXRCXM202361)
  • Sanya Science and Technology Special Fund (2024KJFX034)
  • Youth Fund Project of Hainan Natural Science Foundation (622QN344)
  • National Key Research and Development Program of China (2022ZD0160604)
  • National Natural Science Foundation of China (62176194)
  • Key Research and Development Program of Hubei Province (2023BAB083)
  • Project of Sanya Yazhou Bay Science and Technology City (SCKJ-JYRC-2022-76, SKJC-2022-PTDX-031, SKJC-JYRC-2024-63)
  • National Natural Science Foundation of China (62401415, 62271484)
  • Hubei Natural Science Foundation for Youths (2023AFB146)
  • MindSpore, which is a new deep learning computing framework
  • National Science Fund for Distinguished Young Scholars (61925112)
  • Key Research and Development Projects of Shaanxi Province (2023-YBGY225)
  • High performance Computing Platform of YZBSTCACC

This publication has 43 references indexed in Scilit: