Synthetic-to-Real Translation for Class-Agnostic Motion Prediction
- 12 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP (15209210) , 1-15
Abstract
Motion understanding is critical for ensuring safety and robustness in autonomous driving systems, driving increasing interest in motion prediction. A key challenge in this domain is the high cost associated with acquiring real-world motion labels. It is therefore ideal if we could transfer motion knowledge from synthetic data to real data. In this context, we explore the potential of synthetic-to-real translation for motion prediction (SRMP). However, the most used naive motion regression methods are notably sensitive to the synthetic-to-real domain shift, resulting in unreliable knowledge translation. To address this, we propose a novel approach integrating a motion knowledge translation framework with two key components: (1) objectness-aware motion prediction, which explicitly models the joint distribution of motion patterns and objectness priors to improve domain-invariant feature learning, and (2) objectnessaided motion enhancement, a motion label refinement mechanism that leverages learned objectness priors to filter motion noise. Furthermore, we present a practical pipeline for generating Motion4D, the first synthetic 4D LiDAR dataset tailored for SRMP research, addressing the lack of synthetic motion benchmarks. Experimental results demonstrate that our approach effectively bridges the domain gaps and yields superior performance on real scenes. Code and dataset will be made publicly available.Keywords
All Related Versions
Funding Information
- Agency for Science, Technology and Research
- MTC Programmatic Funds (M23L7b0021)
- MOE AcRF Tier 2 (MOE-T2EP20223-0001)
This publication has 53 references indexed in Scilit:
- AI models collapse when trained on recursively generated dataNature, 2024
- Audio–visual representation learning for anomaly events detection in crowdsNeurocomputing, 2024
- GLENet: Boosting 3D Object Detectors with Generative Label Uncertainty EstimationInternational Journal of Computer Vision, 2023
- 3D Instances as 1D Kernels2022
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset2020
- PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation2020
- LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World2020
- PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume2018
- A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation2016
- Multitask LearningMachine Learning, 1997