A Multimodal 3-D Detector With Attention From the Corresponding Modal
- 8 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 23 (8) , 8581-8590
Abstract
Robust 3D detection based on camera-LiDAR fusion has become a research focus for autonomous driving. Despite the extensive efforts invested, it remains an extremely challenging task to effectively establish fine-grained corresponding relations between different modalities and achieve complementary multimodal fusion free of interference. To these ends, we propose a multimodal 3D detector termed as CFFNet, mainly consisting of a multimodal feature extraction backbone network and a corresponding feature-based attention fusion (CFAF) module. Specifically, the multimodal feature extraction backbone is devised to build a simple yet effective cross-modal correspondence by unifying multimodal features into a shared bird’s eye view (BEV) space. Expressive and noise-suppressed image BEV features are generated from pseudo image voxels, whose depth information is estimated with range-wise attention drawn from LiDAR BEV features. With the prominent heterogeneity of multimodal features taken into consideration, the CFAF module performs a discriminative fusion targeted at informative and contributive features retained in different modalities. The fusion is accomplished by element-wisely reweighting image BEV features with attention generated from differential features of LiDAR. The multimodal feature extraction backbone and CFAF module are both evaluated and proved to offer significant performance gains separately and together. Evaluations conducted on the KITTI 3D object detection dataset show that the proposed CFFNet achieves state-of-the-art performance.Keywords
Funding Information
- National Natural Science Foundation of China (51805388)
- Open Foundation of Foshan Xianhu Laboratory of the Advanced Energy Science and Technology Guangdong Laboratory (XHD2020-003)
This publication has 41 references indexed in Scilit:
- SECOND: Sparsely Embedded Convolutional DetectionSensors, 2018
- Deep Continuous Fusion for Multi-sensor 3D Object Detection2018
- Focal Loss for Dense Object DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- A General Pipeline for 3D Detection of Vehicles2018
- Feature Pyramid Networks for Object Detection2017
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation2017
- Multi-view 3D Object Detection Network for Autonomous Driving2017
- 3D Bounding Box Estimation Using Deep Learning and Geometry2017
- Cyclical Learning Rates for Training Neural Networks2017
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012