Multi-Modal 3D Object Detection by Box Matching
Authors:
- Zhe LiuZLZhe Liu
- Xiaoqing YeXYXiaoqing Ye
- Zhikang ZouZZZhikang Zou
- Xinwei HeXHXinwei He
- Xiao TanXTXiao Tan
- Errui DingEDErrui Ding
- Jingdong WangJWJingdong Wang
- Xiang BaiXBXiang Bai
Preprint
Abstract
Multi-modal 3D object detection has received growing attention as the information from different sensors like LiDAR and cameras are complementary. Most fusion methods for 3D detection rely on an accurate alignment and calibration between 3D point clouds and RGB images. However, such an assumption is not reliable in a real-world self-driving system, as the alignment between different modalities is easily affected by asynchronous sensors and disturbed sensor placement. We propose a novel {F}usion network by {B}ox {M}atching (FBMNet) for multi-modal 3D detection, which provides an alternative way for cross-modal feature alignment by learning the correspondence at the bounding box level to free up the dependency of calibration during inference. With the learned assignments between 3D and 2D object proposals, the fusion for detection can be effectively performed by combing their ROI features. Extensive experiments on the nuScenes dataset demonstrate that our method is much more stable in dealing with challenging cases such as asynchronous sensors, misaligned sensor placement, and degenerated camera images than existing fusion methods. We hope that our FBMNet could provide an available solution to dealing with these challenging cases for safety in real autonomous driving scenarios. Codes will be publicly available at https://github.com/happinesslz/FBMNet.Keywords
All Related Versions
This publication has 55 references indexed in Scilit:
- Deformable Feature Aggregation for Dynamic Multi-modal 3D Object Detection2022
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- PETR: Position Embedding Transformation for Multi-view 3D Object Detection2022
- 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-view Spatial Feature Fusion for 3D Object Detection2020
- EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection2020
- End-to-End Object Detection with Transformers2020
- Deep Continuous Fusion for Multi-sensor 3D Object Detection2018
- Multi-view 3D Object Detection Network for Autonomous Driving2017
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014