Robust-FusionNet: Deep Multimodal Sensor Fusion for 3-D Object Detection Under Severe Weather Conditions
- 18 July 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 71 (00189456) , 1-13
Abstract
The LiDAR point cloud data and camera images are distorted to a different degree under various severe weather conditions. Due to this, the traditional single-modal object detection methods are unable to use the complementary information between different sensors. Consequently, these algorithms are unable to address various issues caused by severe weather conditions. Recently, the multimodal data fusion methods are applied to the road object detection under severe weather conditions. However, the multimodal algorithms suffer from low data alignment accuracy and the inability to suppress the changes in exposure under severe weather conditions. In this work, we propose a new multimodal sensor fusion object detection network. The proposed network effectively overcomes the shortcomings caused by the camera and LiDAR distortions in severe weather conditions and achieves robust environment perception. We propose: 1) pointwise aligned data fusion method based on K-means++ clustering to improve the accuracy of data alignment; 2) implicit feature pyramid network (i-FPN) to fuse the image features for suppressing the distortions caused by the changes in exposure; and 3) hybrid attention mechanism (HAM) to deal with the fusion features and improve the adaptability toward different working conditions. We perform experiments using the One Million Scenes for Autonomous Driving (ONCE) and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) datasets. The experimental results and analysis show that the proposed method effectively improves the performance of multimodal deep fusion network under both clear and severe weather conditions.Keywords
Funding Information
- National Natural Science Foundation of China (U20A20333, 52072160, 51875255)
- Key Research and Development Program of Jiangsu Province (BE2019010-2, BE2020083-3)
- Natural Science Foundation of Jiangsu Province (BK20190853)
- Jiangsu Province’s six talent peaks (TD-GDZB-022)
- Australian Research Council (ARC) Discovery Early Career Researcher Award (DECRA), Australia (DE190100931)
This publication has 43 references indexed in Scilit:
- SECOND: Sparsely Embedded Convolutional DetectionSensors, 2018
- Deep Continuous Fusion for Multi-sensor 3D Object Detection2018
- Frustum PointNets for 3D Object Detection from RGB-D Data2018
- 3-D LiDAR + Monocular Camera: An Inverse-Depth-Induced Fusion Framework for Urban Road DetectionIEEE Transactions on Intelligent Vehicles, 2018
- GOMSF: Graph-Optimization Based Multi-Sensor Fusion for robust UAV Pose estimation2018
- Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems2018
- Feature Pyramid Networks for Object Detection2017
- Multi-view 3D Object Detection Network for Autonomous Driving2017
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- Multi-sensor optimal information fusion Kalman filterAutomatica, 2004