Self-Supervised 3D Scene Flow Estimation and Motion Prediction Using Local Rigidity Prior
- 14 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (12) , 8106-8122
Abstract
In this article, we investigate self-supervised 3D scene flow estimation and class-agnostic motion prediction on point clouds. A realistic scene can be well modeled as a collection of rigidly moving parts, therefore its scene flow can be represented as a combination of rigid motion of these individual parts. Building upon this observation, we propose to generate pseudo scene flow labels for self-supervised learning through piecewise rigid motion estimation, in which the source point cloud is decomposed into local regions and each region is treated as rigid. By rigidly aligning each region with its potential counterpart in the target point cloud, we obtain a region-specific rigid transformation to generate its pseudo flow labels. To mitigate the impact of potential outliers on label generation, when solving the rigid registration for each region, we alternately perform three steps: establishing point correspondences, measuring the confidence for the correspondences, and updating the rigid transformation based on the correspondences and their confidence. As a result, confident correspondences will dominate label generation, and a validity mask will be derived for the generated pseudo labels. By using the pseudo labels together with their validity mask for supervision, models can be trained in a self-supervised manner. Extensive experiments on FlyingThings3D and KITTI datasets demonstrate that our method achieves new state-of-the-art performance in self-supervised scene flow learning, without any ground truth scene flow for supervision, even performing better than some supervised counterparts. Additionally, our method is further extended to class-agnostic motion prediction and significantly outperforms previous state-of-the-art self-supervised methods on nuScenes dataset.Keywords
All Related Versions
This publication has 71 references indexed in Scilit:
- 3D Object Detection with a Self-supervised Lidar Scene Flow Backbone2022
- What Matters for 3D Scene Flow Network2022
- FH-Net: A Fast Hierarchical Network for Scene Flow Estimation on Real-World Point Clouds2022
- Dynamic 3D Scene Analysis by Point Cloud Accumulation2022
- Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving2022
- Bi-PointFlowNet: Bidirectional Learning for Point Cloud Based Scene Flow Estimation2022
- FLOT: Scene Flow on Point Clouds Guided by Optimal Transport2020
- PointPWC-Net: Cost Volume on Point Clouds for (Self-)Supervised Scene Flow Estimation2020
- Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation2018
- Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow2010