$\mathbf{C}^2$Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection
Preprint
Abstract
Object detection on visible (RGB) and infrared (IR) images, as an emerging solution to facilitate robust detection for around-the-clock applications, has received extensive attention in recent years. With the help of IR images, object detectors have been more reliable and robust in practical applications by using RGB-IR combined information. However, existing methods still suffer from modality miscalibration and fusion imprecision problems. Since transformer has the powerful capability to model the pairwise correlations between different features, in this paper, we propose a novel Calibrated and Complementary Transformer called $\mathrm{C}^2$Former to address these two problems simultaneously. In $\mathrm{C}^2$Former, we design an Inter-modality Cross-Attention (ICA) module to obtain the calibrated and complementary features by learning the cross-attention relationship between the RGB and IR modality. To reduce the computational cost caused by computing the global attention in ICA, an Adaptive Feature Sampling (AFS) module is introduced to decrease the dimension of feature maps. Because $\mathrm{C}^2$Former performs in the feature domain, it can be embedded into existed RGB-IR object detectors via the backbone network. Thus, one single-stage and one two-stage object detector both incorporating our $\mathrm{C}^2$Former are constructed to evaluate its effectiveness and versatility. With extensive experiments on the DroneVehicle and KAIST RGB-IR datasets, we verify that our method can fully utilize the RGB-IR complementary information and achieve robust detection results. The code is available at https://github.com/yuanmaoxun/Calibrated-and-Complementary-Transformer-for-RGB-Infrared-Object-Detection.git.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions2024
- Improving RGB-infrared object detection with cascade alignment-guided transformerInformation Fusion, 2024
- Cross-modality attentive feature fusion for object detection in multispectral remote sensing imageryPattern Recognition, 2022
- Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting2021
- Ship detection and classification from optical remote sensing images: A surveyChinese Journal of Aeronautics, 2021
- RGB-T object tracking: Benchmark and baselinePattern Recognition, 2019
- Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detectionInformation Fusion, 2019
- Cross-modality interactive attention network for multispectral pedestrian detectionInformation Fusion, 2019
- Illumination-aware faster R-CNN for robust multispectral pedestrian detectionPattern Recognition, 2019
- Adaptive deconvolutional networks for mid and high level feature learning2011