AMMF: Attention-Based Multi-Phase Multi-Task Fusion for Small Contour Object 3D Detection
- 16 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. PP (15249050) , 1-10
Abstract
Recently significant progress has been made in 3D detection. However, it is still challenging to detect small contour objects under complex scenes. This paper proposes a novel Attention-based Multi-phase Multi-task Fusion (AMMF) that uses point-level, RoI-level, and multi-task fusions to complement the disadvantages of LiDAR and camera, to solve this challenge. First, at the feature extraction phase, AMMF uses the Low and High-level Fusion with Matching Attention (LHF-MA) and efficient FPN (eFPN) to perform point-level fusion for cross sensors and single sensor, respectively. Instead of merging each level and using expensive 3D CNN like other methods, LHF-MA fuses low-level spatial location and high-level contextual feature of 2D CNN customized feature extractors and ignores the fusion of middle levels, reducing the computational cost. Then, at the proposal generation phase, Progressive Proposal Fusion (PPF) with learned attention map is used to perform coarse-to-fine RoI-level fusion, instead of only combining coarse-grained features at high-level of network. PPF using progressively increasing IoU thresholds could avoid overfitting and improve the performance. Note that the matching attentions and learned attention maps are utilized to weigh the priority of different sensors. Moreover, to solve the sparseness of point-wise fusion between LiDAR BEV and RGB image, AMMF uses multi-task fusion that generates pseudo-LiDAR from camera by depth estimation task, to guide this point-wise fusion. Finally, AMMF performs excellently for detecting small contour objects like pedestrians, cyclists, and distant cars. On the KITTI, AMMF finishes 3.62% improvements in the moderate instance for pedestrians. It achieves a 2.21% improvement in the $>$ 50 instance of LEVEL $_{-}$ 2 level for vehicle on the Waymo Open Dataset. And AMMF is further verified on our customized dataset consisting of challenging scenarios like strong illumination and heavy shadow cases.Keywords
Funding Information
- International Training Program for Outstanding Young Scientific Research Talents in 2019 of Guangdong Provincial Department of Education
This publication has 25 references indexed in Scilit:
- Stabilization and Validation of 3D Object Position Using Multimodal Sensor Fusion and Semantic SegmentationSensors, 2020
- Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection2019
- GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond2019
- Robust Multi-Modality Multi-Object Tracking2019
- 3D Object Detection Using Scale Invariant and Feature Reweighting NetworksProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Multi-Task Multi-Sensor Fusion for 3D Object Detection2019
- Deep Continuous Fusion for Multi-sensor 3D Object Detection2018
- SBNet: Sparse Blocks Network for Fast Inference2018
- Non-local Neural Networks2018
- Deep Residual Learning for Image Recognition2016