Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection
Preprint
Abstract
In frame-based vision, object detection faces substantial performance degradation under challenging conditions due to the limited sensing capability of conventional cameras. Event cameras output sparse and asynchronous events, providing a potential solution to solve these problems. However, effectively fusing two heterogeneous modalities remains an open issue. In this work, we propose a novel hierarchical feature refinement network for event-frame fusion. The core concept is the design of the coarse-to-fine fusion module, denoted as the cross-modality adaptive feature refinement (CAFR) module. In the initial phase, the bidirectional cross-modality interaction (BCI) part facilitates information bridging from two distinct sources. Subsequently, the features are further refined by aligning the channel-level mean and variance in the two-fold adaptive feature refinement (TAFR) part. We conducted extensive experiments on two benchmarks: the low-resolution PKU-DDD17-Car dataset and the high-resolution DSEC dataset. Experimental results show that our method surpasses the state-of-the-art by an impressive margin of $\textbf{8.0}\%$ on the DSEC dataset. Besides, our method exhibits significantly better robustness (\textbf{69.5}\% versus \textbf{38.7}\%) when introducing 15 different corruption types to the frame images. The code can be found at the link (https://github.com/HuCaoFighting/FRN).
Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Specificity-preserving RGB-D saliency detectionComputational Visual Media, 2023
- Event-Based Fusion for Motion Deblurring with Cross-modal Attention2022
- UCTNet: Uncertainty-Aware Cross-Modal Transformer Network for Indoor RGB-D Semantic Segmentation2022
- VCANet: Vanishing-Point-Guided Context-Aware Network for Small Road Object DetectionAutomotive Innovation, 2021
- Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation2020
- Learning to Exploit Multiple Vision Modalities by Using Grafted Networks2020
- Cascade Graph Neural Networks for RGB-D Salient Object Detection2020
- Deep Learning for Generic Object Detection: A SurveyInternational Journal of Computer Vision, 2019
- CBAM: Convolutional Block Attention Module2018
- Microsoft COCO: Common Objects in Context2014