A Collaborative Error-Tolerant Synchronization Framework With its Application in Multimodal Light Field System Development
- 25 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 25 (21) , 40512-40524
Abstract
Multimodal systems integrating cameras, LiDARs, radars, and other sensors are increasingly essential for intelligent transportation due to their superior environmental perception. However, the lack of a unified synchronization mechanism across diverse sensor modalities poses challenges for system-wide synchronization, causing data misalignment, reduced acquisition frequency, and degraded perception accuracy. This paper proposes a novel multimodal synchronization framework that enables high-frequency and low-latency data acquisition to improve perception accuracy. The framework integrates multi-frequency hardware synchronization with a temporal-spatial error-tolerant model to minimize data misalignment while maximizing effective frame rates. A frame header-based time synchronizer generates corrected multi-frequency trigger signals, achieving sub-4ms synchronization across 12 sensors and 4 modalities, improving the mean Intersection over Union (mIOU) of dynamic objects by an average of 4.97% in point cloud projection and 15.50% in image registration compared to existing methods. It enables us to build a multimodal system with the highest modality count (a 3×3 camera array, a LiDAR, a radar, and a long-wave infrared camera) and frame rate, capable of capturing novel light field data, outperforming existing multimodal setups. Evaluations on our real-world dataset show Average Precision (AP) gains of 49.35% for cyclists and 39.53% for pedestrians in object detection, along with significant improvements in image translation.Keywords
Funding Information
- Shenzhen Project (JSGG20220831095602005)
- Natural Science Foundation of Guangdong Province, China (2023A1515012716)
This publication has 43 references indexed in Scilit:
- Multi-Modal 3D Object Detection in Autonomous Driving: A SurveyInternational Journal of Computer Vision, 2023
- Enhancing Grid-Based 3D Object Detection in Autonomous Driving With Improved Dimensionality ReductionIEEE Access, 2023
- PillarNet: Real-Time and High-Performance Pillar-Based 3D Object Detection2022
- Improvements to Target-Based 3D LiDAR to Camera CalibrationIEEE Access, 2020
- PointPainting: Sequential Fusion for 3D Object Detection2020
- Semantic Image Synthesis With Spatially-Adaptive Normalization2019
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs2018
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Globally consistent depth labeling of 4D light fields2012
- Synthetic Aperture Focusing using a Shear-Warp Factorization of the Viewing Transform2006