Cross-modality feature fusion for night pedestrian detection
Authors:
- Yong FengYFYong Feng
- Enbo LuoELEnbo Luo
- Hai LuHLHai Lu
- SuWei ZhaiSZSuWei Zhai
Open Access
- 26 March 2024
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Physics
- Vol. 12, 1356248
Abstract
Night pedestrian detection with visible image only suffers from the dilemma of high miss rate due to poor illumination conditions. Cross-modality fusion can ameliorate this dilemma by providing complementary information to each other through infrared and visible images. In this paper, we propose a cross-modal fusion framework based on YOLOv5, which is aimed at addressing the challenges of night pedestrian detection under low-light conditions. The framework employs a dual-stream architecture that processes visible images and infrared images separately. Through the Cross-Modal Feature Rectification Module (CMFRM), visible and infrared features are finely tuned on a granular level, leveraging their spatial correlations to focus on complementary information and substantially reduce uncertainty and noise from different modalities. Additionally, we have introduced a two-stage Feature Fusion Module (FFM), with the first stage introducing a cross-attention mechanism for cross-modal global reasoning, and the second stage using a mixed channel embedding to produce enhanced feature outputs. Moreover, our method involves multi-dimensional interaction, not only correcting feature maps in terms of channel and spatial dimensions but also applying cross-attention at the sequence processing level, which is critical for the effective generalization of cross-modal feature combinations. In summary, our research significantly enhances the accuracy and robustness of nighttime pedestrian detection, offering new perspectives and technical pathways for visual information processing in low-light environments.Keywords
This publication has 67 references indexed in Scilit:
- A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict RegistrationInternational Journal of Computer Vision, 2023
- Multimodal Object Detection via Probabilistic Ensembling2022
- Human Action Recognition and Prediction: A SurveyInternational Journal of Computer Vision, 2022
- Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems2020
- Multi-focus image fusion: A Survey of the state of the artInformation Fusion, 2020
- Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detectionInformation Fusion, 2019
- Cross-modality interactive attention network for multispectral pedestrian detectionInformation Fusion, 2019
- Occlusion-Aware R-CNN: Detecting Pedestrians in a Crowd2018
- Intelligent Video Monitoring System with the Functionality of Online Recognition of People’s Behavior and Interactions Between People2018
- SSD: Single Shot MultiBox Detector2016