Vehicle Detection Based on Adaptive Multimodal Feature Fusion and Cross-Modal Vehicle Index Using RGB-T Images
Open Access
- 12 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 16 (19391404) , 8166-8177
Abstract
Target detection is a critical task in interpreting aerial images. Small target detection, such as vehicles, is challenging. Different lighting conditions affect the accuracy of vehicle detection. For example, vehicles are difficult to distinguish from the background in RGB images under low illumination conditions. In contrast, under high illumination conditions, the color and texture of vehicles are not significantly different in thermal infrared (TIR) images. To improve the accuracy of vehicle detection under various illumination conditions, we propose an adaptive multi-modal feature fusion and cross-modal vehicle index (AFFCM) model for vehicle detection. Based on the single-stage object detection model, AFFCM uses red, green, blue, and thermal infrared (RGB-T) images. It comprises three parts: 1) the softpooling channel attention (SCA) mechanism calculates the cross-modal feature weights of the RGB and TIR features using a fully connected layer during global weighted pooling. 2) We design a multi-modal adaptive feature fusion (MAFF) module based on the cross-modal feature weights derived from the SCA mechanism. The MAFF selects features with high weight, compresses redundant features with low weight, and performs adaptive fusion using a multi-scale feature pyramid. 3) A cross-modal vehicle index is established to extract the target area, suppress complex background information, and minimize false alarms in vehicle detection. The mean average precision (mAP) on the Drone Vehicle dataset are 14.44% and 5.02% higher than those obtained using only RGB or TIR images. The mAP is 2.63% higher than that of state-of-the-art (SOTA) methods that utilize RGB and TIR images.Keywords
Funding Information
- National Key R&D Program of China (2021YFA0715203)
- National Natural Science Foundation of China (62001455, 41871245)
This publication has 43 references indexed in Scilit:
- A survey on object detection in optical remote sensing imagesISPRS Journal of Photogrammetry and Remote Sensing, 2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Multispectral Deep Neural Networks for Pedestrian Detection2016
- Fast R-CNN2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Analysis on Development Status and Trend of Vehicle Materials YourAdvanced Materials Research, 2013
- Old Asphalt Road Surface Mixture of Cold Regeneration Research Laboratory TestApplied Mechanics and Materials, 2012
- On the Specific Heat Capacity of CuO NanofluidAdvances in Mechanical Engineering, 2010
- A Threshold Selection Method from Gray-Level HistogramsIEEE Transactions on Systems, Man, and Cybernetics, 1979