OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving
Top Cited Papers
- 11 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (6) , 7032-7049
Abstract
The rapid advancement of deep learning has intensified the need for comprehensive data for use by autonomous driving algorithms. High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions. Next-generation autonomous driving datasets must be multimodal, incorporating data from advanced sensors that feature extensive data coverage, detailed annotations, and diverse scene representation. To address this need, we present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achieve full environmental perception. The dataset comprises 1501 clips, each approximately 30-s long, totaling more than 450 K synchronized frames and more than 5.85 million synchronized sensor data points. We also propose a novel 4D annotation pipeline. To date, we have annotated 200 clips with more than 514 K precise 3D bounding boxes. These clips also include semantic segmentation annotations for static scene elements. Additionally, we introduce a novel automated pipeline for generation of the dense occupancy ground truth, which effectively leverages information from non-key frames. Alongside the proposed dataset, we establish comprehensive evaluation metrics, baseline models, and benchmarks for 3D detection and semantic occupancy prediction. These benchmarks utilize surround-view cameras and 4D imaging radar to explore cost-effective sensor solutions for autonomous driving applications. Extensive experiments demonstrate the effectiveness of our low-cost sensor configuration and its robustness under adverse conditions.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2022YFB2503404)
This publication has 52 references indexed in Scilit:
- Dual Radar: A Multi-modal Dataset with Dual 4D Radar for Autononous DrivingScientific Data, 2025
- OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving2024
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- OpenCalib: A multi-sensor calibration toolbox for autonomous drivingSoftware Impacts, 2022
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset2020
- YOLO-face: a real-time face detectorThe Visual Computer, 2020
- Occupancy Networks: Learning 3D Reconstruction in Function Space2019
- PointPillars: Fast Encoders for Object Detection From Point Clouds2019
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012