CoFiI2P: Coarse-to-Fine Correspondences-Based Image to Point Cloud Registration
- 23 September 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (11) , 10264-10271
Abstract
Image-to-point cloud (I2P) registration is a fundamental task for robots and autonomous vehicles to achieve cross-modality data fusion and localization. Current I2P registration methods primarily focus on estimating correspondences at the point or pixel level, often neglecting global alignment. As a result, I2P matching can easily converge to a local optimum if it lacks high-level guidance from global constraints. To improve the success rate and general robustness, this letter introduces CoFiI2P, a novel I2P registration network that extracts correspondences in a coarse-to-fine manner. First, the image and point cloud data are processed through a two-stream encoder-decoder network for hierarchical feature extraction. Second, a coarse-to-fine matching module is designed to leverage these features and establish robust feature correspondences. Specifically, in the coarse matching phase, a novel I2P transformer module is employed to capture both homogeneous and heterogeneous global information from the image and point cloud data. This enables the estimation of coarse super-point/super-pixel matching pairs with discriminative descriptors. In the fine matching module, point/pixel pairs are established with the guidance of super-point/super-pixel correspondences. Finally, based on matching pairs, the transformation matrix is estimated with the EPnP-RANSAC algorithm. Experiments conducted on the KITTI Odometry dataset demonstrate that CoFiI2P achieves impressive results, with a relative rotation error (RRE) of 1.14 degrees and a relative translation error (RTE) of 0.29 meters, while maintaining real-time speed. These results represent a significant improvement of 84% in RRE and 89% in RTE compared to the current state-of-the-art (SOTA) method. Additional experiments on the Nuscenes dataset confirm our method's generalizability.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (42201477, 42130105)
- Open Fund of Hubei Luojia Laboratory (2201000054)
- Open Fund of Key Laboratory of Urban Spatial Information
- Ministry of Natural Resources (2023ZD001)
This publication has 37 references indexed in Scilit:
- HCTO: Optimality-aware LiDAR inertial odometry with hybrid continuous time optimization for compact wearable mapping systemISPRS Journal of Photogrammetry and Remote Sensing, 2024
- 2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds2022
- Learning Cross-Domain Descriptors for 2D-3D Matching with Hard Triplet Loss and Spatial Transformer Network2021
- The Perfect Match: 3D Point Cloud Matching With Smoothed Densities2019
- PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors2018
- Automatic registration of panoramic image sequence and mobile laser scanning data using semantic featuresISPRS Journal of Photogrammetry and Remote Sensing, 2018
- Deep Residual Learning for Image Recognition2016
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012
- EPnP: An Accurate O(n) Solution to the PnP ProblemInternational Journal of Computer Vision, 2008
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004