SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection
Authors:
- Shuhan DongSDShuhan Dong
- Yunsong LiYunsong Li
- Weiying XieWXWeiying Xie
- Jiaqing ZhangJZJiaqing Zhang
- Jiayuan TianJTJiayuan Tian
- Danian YangDYDanian Yang
- Jie LeiJLJie Lei
Preprint
Abstract
Multimodal object detection leverages diverse modal information to enhance the accuracy and robustness of detectors. By learning long-term dependencies, Transformer can effectively integrate multimodal features in the feature extraction stage, which greatly improves the performance of multimodal object detection. However, current methods merely stack Transformer-guided fusion techniques without exploring their capability to extract features at various depth layers of network, thus limiting the improvements in detection performance. In this paper, we introduce an accurate and efficient object detection method named SeaDATE. Initially, we propose a novel dual attention Feature Fusion (DTF) module that, under Transformer's guidance, integrates local and global information through a dual attention mechanism, strengthening the fusion of modal features from orthogonal perspectives using spatial and channel tokens. Meanwhile, our theoretical analysis and empirical validation demonstrate that the Transformer-guided fusion method, treating images as sequences of pixels for fusion, performs better on shallow features' detail information compared to deep semantic information. To address this, we designed a contrastive learning (CL) module aimed at learning features of multimodal samples, remedying the shortcomings of Transformer-guided fusion in extracting deep semantic features, and effectively utilizing cross-modal information. Extensive experiments and ablation studies on the FLIR, LLVIP, and M3FD datasets have proven our method to be effective, achieving state-of-the-art detection performance.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Dual feature extraction network for hyperspectral image analysisPattern Recognition, 2021
- A robust and fast multispectral pedestrian detection deep networkKnowledge-Based Systems, 2021
- Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems2020
- Multi-modal Transformer for Video Retrieval2020
- Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks2020
- Deep Learning for Generic Object Detection: A SurveyInternational Journal of Computer Vision, 2019
- Cross-modality interactive attention network for multispectral pedestrian detectionInformation Fusion, 2019
- Infrared and visible image fusion methods and applications: A surveyInformation Fusion, 2019
- Deep Multimodal Learning: A Survey on Recent Advances and TrendsIEEE Signal Processing Magazine, 2017
- History highlights and future trends of infrared sensorsJournal of Modern Optics, 2010