Cross Teaching-Enhanced Multispectral Remote Sensing Object Detection With Transformer
Open Access
- 27 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 18 (19391404) , 2401-2413
Abstract
Remote sensing platforms are often equipped with sensors of multiple spectrums to capture the diverse reflective properties of ground areas, typically including the visible spectrum and the near infrared (NIR) spectrum. Moreover, thermal infrared (TIR) sensors capture the radiated heat of targets and are capable of all-day observation regardless of illumination conditions. By leveraging the complementary features of different spectrums, multi-spectral fusion techniques enhance the precision and robustness of remote sensing object detection methods. In this article, we present an object detection method for remote sensing imagery named Multi-spectral Detection Transformer (Multi-spectral DETR). The model fuses multi-spectral features with deformable attention and utilizes fused features for object detection. The Multi-spectral Deformable Attention (MDA) fusion block integrates the flexibility of dynamic weights with the principle of fusion based on local regions. Then, we propose a simple yet effective oriented object detection scheme based on angle prediction. Finally, we introduce a novel cross-teaching method between single-spectral and multi-spectral models, which alleviates the spectral interference issue caused by inconsistent target visibility. Experimental results demonstrate that Multi-spectral DETR achieves state-of-the-art results on both the RGB-NIR VEDAI and the RGB-TIR DroneVehicle datasets.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFFO602103)
- National Key Research and Development Program of China (2021YFF0602102)
- Shanghai Aerospace Science and Technology Innovation Foundation (SAST2019-048)
This publication has 43 references indexed in Scilit:
- ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detectionPattern Recognition, 2023
- Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection2022
- Cross-modality attentive feature fusion for object detection in multispectral remote sensing imageryPattern Recognition, 2022
- On the Arbitrary-Oriented Object Detection: Classification Based Approaches RevisitedInternational Journal of Computer Vision, 2022
- Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems2020
- Arbitrary-Oriented Object Detection with Circular Smooth Label2020
- End-to-End Object Detection with Transformers2020
- IoU Loss for 2D/3D Object Detection2019
- Vehicle detection in aerial imagery : A small target detection benchmarkJournal of Visual Communication and Image Representation, 2016
- Landsat-8: Science and product vision for terrestrial global change researchRemote Sensing of Environment, 2014