A Collaborative Error-Tolerant Synchronization Framework With its Application in Multimodal Light Field System Development

Authors:

Abstract
Multimodal systems integrating cameras, LiDARs, radars, and other sensors are increasingly essential for intelligent transportation due to their superior environmental perception. However, the lack of a unified synchronization mechanism across diverse sensor modalities poses challenges for system-wide synchronization, causing data misalignment, reduced acquisition frequency, and degraded perception accuracy. This paper proposes a novel multimodal synchronization framework that enables high-frequency and low-latency data acquisition to improve perception accuracy. The framework integrates multi-frequency hardware synchronization with a temporal-spatial error-tolerant model to minimize data misalignment while maximizing effective frame rates. A frame header-based time synchronizer generates corrected multi-frequency trigger signals, achieving sub-4ms synchronization across 12 sensors and 4 modalities, improving the mean Intersection over Union (mIOU) of dynamic objects by an average of 4.97% in point cloud projection and 15.50% in image registration compared to existing methods. It enables us to build a multimodal system with the highest modality count (a 3×3 camera array, a LiDAR, a radar, and a long-wave infrared camera) and frame rate, capable of capturing novel light field data, outperforming existing multimodal setups. Evaluations on our real-world dataset show Average Precision (AP) gains of 49.35% for cyclists and 39.53% for pedestrians in object detection, along with significant improvements in image translation.
Funding Information
  • Shenzhen Project (JSGG20220831095602005)
  • Natural Science Foundation of Guangdong Province, China (2023A1515012716)