Boosting Instance Awareness Via Cross-View Correlation With 4D Radar and Camera for 3D Object Detection
- 17 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-13
Abstract
4D millimeter-wave radar has emerged as a promising sensing modality for autonomous driving due to its robustness and affordability. However, its sparse and weak geometric cues make reliable instance activation difficult, limiting the effectiveness of existing radar–camera fusion paradigms. BEV-level fusion offers global scene understanding but suffers from weak instance focus, while perspective-level fusion captures instance details but lacks holistic context. To address these limitations, we propose SIFormer, a scene–instance aware transformer for 3D object detection using 4D radar and camera. SIFormer first suppresses background noise during view transformation through segmentation- and depth-guided localization. It then introduces a cross-view activation mechanism that injects 2D instance cues into BEV space, enabling reliable instance awareness under weak radar geometry. Finally, a transformer-based fusion module aggregates complementary image semantics and radar geometry for robust perception. As a result, with the aim of enhancing instance awareness, SIFormer bridges the gap between the two paradigms, combining their complementary strengths to address inherent sparse nature of radar and improve detection accuracy. Experiments demonstrate that SIFormer achieves state-of-the-art performance on View-of-Delft, TJ4DRadSet and NuScenes datasets. The code will be made publicly available.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2023YFB3209800)
This publication has 46 references indexed in Scilit:
- 3D Object Detection for Autonomous Driving: A Comprehensive SurveyInternational Journal of Computer Vision, 2023
- Perception and sensing for autonomous vehicles under adverse weather conditions: A surveyISPRS Journal of Photogrammetry and Remote Sensing, 2023
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- PointPainting: Sequential Fusion for 3D Object Detection2020
- PointPillars: Fast Encoders for Object Detection From Point Clouds2019
- Cascade R-CNN: Delving Into High Quality Object Detection2018
- Microsoft COCO: Common Objects in Context2014
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012