Unifying Obstacle Detection, Recognition, and Fusion Based on the Polarization Color Stereo Camera and LiDAR for the ADAS
Open Access
Abstract
The perception module plays an important role in vehicles equipped with advanced driver-assistance systems (ADAS). This paper presents a multi-sensor data fusion system based on the polarization color stereo camera and the forward-looking light detection and ranging (LiDAR), which achieves the multiple target detection, recognition, and data fusion. The You Only Look Once v4 (YOLOv4) network is utilized to achieve object detection and recognition on the color images. The depth images are obtained from the rectified left and right images based on the principle of the epipolar constraints, then the obstacles are detected from the depth images using the MeanShift algorithm. The pixel-level polarization images are extracted from the raw polarization-grey images, then the water hazards are detected successfully. The PointPillars network is employed to detect the objects from the point cloud. The calibration and synchronization between the sensors are accomplished. The experiment results show that the data fusion enriches the detection results, provides high-dimensional perceptual information and extends the effective detection range. Meanwhile, the detection results are stable under diverse range and illumination conditions.Keywords
Funding Information
- National Natural Science Foundation of China (U21B6001, 62020106002, 61735017, 61822510)
- National Key Basic Research Program of China (2021YFC2401403)
- Major Scientific Research Project of Zhejiang Laboratory (2019MC0AD02)
This publication has 35 references indexed in Scilit:
- Object Detection With Deep Learning: A ReviewIEEE Transactions on Neural Networks and Learning Systems, 2019
- LIDAR–camera fusion for road detection using fully convolutional neural networksRobotics and Autonomous Systems, 2019
- Frustum PointNets for 3D Object Detection from RGB-D Data2018
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection2018
- A simple, robust and fast method for the perspective-n-point ProblemPattern Recognition Letters, 2018
- Detecting Traversable Area and Water Hazards for the Visually Impaired with a pRGB-D SensorSensors, 2017
- Deep learningNature, 2015
- Track-to-Track Fusion With Asynchronous Sensors Using Information Matrix Fusion for Surround Environment PerceptionIEEE Transactions on Intelligent Transportation Systems, 2012
- Review of Stereo Vision Algorithms: From Software to HardwareInternational Journal of Optomechatronics, 2008
- A flexible new technique for camera calibrationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000