Dongkyu Lee

This paper addresses a novel camera-LiDAR online targetless calibration method. As the installation of heterogeneous sensors in autonomous vehicles is increasing, the importance of sensor calibration is also growing. An automatic sensor calibration function is essential for safe autonomous driving, responding to the sensor's geometric change. In this paper, background knowledge of sensor calibration is explained, and related papers are referenced. Depth estimation and semantic segmentation models are learned to detect significant regions from the camera using open and custom datasets. This paper describes the online calibration procedures based on motion-based calibration between the camera and LiDAR. Motion-based calibration faces a challenge for lack of roll and pitch variation, and our method overcomes this challenge by estimating a camera 3D ground plane from semantic segmentation and depth estimation and simultaneously detecting a 3D LiDAR ground plane. LiDAR-to-camera roll, pitch, and z values are then extracted using the camera and LiDAR ground plane, and the motion-based calibration problem is solved and optimized using those values as constraints. Experimental results conducted on urban roads and the proving ground C-track using our autonomous vehicles showed that the proposed method is quantitatively and qualitatively improving the existing method

In autonomous driving, the camera and LiDAR are complementary to each other through their strengths, and many autonomous vehicles and robot recognition systems utilize cameras and LiDAR. In order to effectively fuse the data of the camera and LiDAR attached to different positions and angles, we must perform camera–LiDAR extrinsic calibration. Most existing camera–LiDAR calibration methods infer the results by constructing a camera–LiDAR feature pair using features acquired from a single frame. It has the disadvantage that it is challenging to draw results. In this paper, we used sequential LiDAR data to extract features by accumulating LiDAR frames effectively. By using the location information between the accumulated frame (global) and the single frame (local) to convert the features detected from the global to the local, the shortcomings of LiDAR’s single frame with few features were supplemented. Methods for detecting feature points in LiDAR are described step-by-step, and the advantages and excellence of this camera–LiDAR calibration system are shown through quantitative/qualitative evaluation methods. As a result, the proposed system outputs superior performance compared to other systems.

In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that holds inter-class relations which send a meaningful supervision to a student; hence, much effort has been put to find such knowledge to be distilled. In this paper, we explore a question that has been given little attention: "when to distill such knowledge." The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscalibration of a student. This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data. We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level. Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model generalization, but also significantly lowers model calibration error.

In this paper, I propose a bird eye view image detection method for parking areas and collision risk areas at the same time in parking situations. Deep learning algorithms using area detection and semantic segmentation were used. The main architecture of the method described in this paper is based on a harmonic densely connected network and a cross-stage partial network. The dataset used for training was calibrated to four 190° wide-angle cameras to generate around view monitor (AVM) images based on the Chungbuk National University parking lot, and an experiment was performed based on this dataset. In the experimental results, the available parking area was visualized by detecting the parking line, parking area, and available driving area in the AVM images. Furthermore, the undetected area in the semantic segmentation as a collision risk area was visualized in order to obtain the results. According to the proposed attention CSPHarDNet model, the experimental results were 81.89% mIoU and 18.36 FPS in a NVIDIA Xavier environment. The results of this experiment demonstrated that algorithms can be used in real time in a parking situation and have better performance results compared to the conventional HarDNet.

Although numerous road segmentation studies have utilized vision data, obtaining robust classification is still challenging due to vision sensor noise and target object deformation. Long-distance images are still problematic because of blur and low resolution, and these features make distinguishing roads from objects difficult. This study utilizes light detection and ranging (LiDAR), which generates information that camera images lack, such as distance, height, and intensity, as a reliable supplement to address this problem. In contrast to conventional approaches, additional domain transformation to a bird’s eye view space is executed to obtain long-range data with resolutions comparable to those of short-range data. This study proposes a convolutional neural network architecture that processes data transformed to a bird’s eye view plane. The network’s pathways are split into two parts to resolve calibration errors in the transformed image and point cloud. The network, which has modules that operate sequentially at various scaled dilated convolution rates, is designed to quickly and accurately handle a wide range of data. Comprehensive empirical studies using the Karlsruhe Institute of Technology and Toyota Technological Institute’s (KITTI’s) road detection benchmarks demonstrate that this study’s approach takes advantage of camera and LiDAR information, achieving robust road detection with short runtimes. Our result ranks 22nd in the KITTI’s leaderboard and shows real-time performance.

This paper proposes a detection method for parking areas and collision risk areas in parking situations. Deep learning algorithms used for area detection rely on semantic segmentation, a method of classifying pixels into semantic segmentation. The main architecture of this paper is based on a harmonic densely connected network and a cross-stage partial network. The dataset was calibrated for four 190-degree wide-angle cameras to generate 500 AVM images based on the Chungbuk National University parking lot, and the experiment was performed with this dataset. In the experimental results, we found that the available parking area can be visualized by detecting the parking line, the parking area, and the available driving area in the AVM image. We visualized the area that remained undetected, as a collision risk area, through semantic segmentation to derive the results. According to the proposed CSPHarDNet model, the experimental results obtained were 81.89% mIoU and 18.36 FPS in the NVIDIA Xavier environment.

In this work, a study was carried out to estimate a look-up table (LUT) that converts a camera image plane to a birds eye view (BEV) plane using a single camera. The traditional camera pose estimation fields require high costs in researching and manufacturing autonomous vehicles for the future and may require pre-configured infra. This paper proposes an autonomous vehicle driving camera calibration system that is low cost and utilizes low infra. A network that outputs an image in the form of an LUT that converts the image into a BEV by estimating the camera pose under urban road driving conditions using a single camera was studied. We propose a network that predicts human-like poses from a single image. We collected synthetic data using a simulator, made BEV and LUT as ground truth, and utilized the proposed network and ground truth to train pose estimation function. In the progress, it predicts the pose by deciphering the semantic segmentation feature and increases its performance by attaching a layer that handles the overall direction of the network. The network outputs camera angle (roll/pitch/yaw) on the 3D coordinate system so that the user can monitor learning. Since the network’s output is a LUT, there is no need for additional calculation, and real-time performance is improved.