MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 21819-21830
Abstract
Vision-language generative AI has demonstrated re-markable promise for empowering cross-modal scene understanding of autonomous driving and high-definition (HD) map systems. However, current benchmark datasets lack multi-modal point cloud, image, and language data pairs. Recent approaches utilize visual instruction learning and cross-modal prompt engineering to expand vision-language models into this domain. In this paper, we pro-pose a new vision-language benchmark that can be used to finetune traffic and HD map domain-specific foundation models. Specifically, we annotate and leverage large-scale, broad-coverage traffic and map data extracted from huge HD map annotations, and use CLIP and LLaMA-2 / Vi-cuna to finetune a baseline model with instruction-following data. Our experimental results across various algorithms reveal that while visual instruction-tuning large language models (LLMs) can effectively learn meaningful represen-tations from MAPLM-QA, there remains significant room for further advancements. To facilitate applying LLMs and multi-modal data into self-driving research, we will release our visual-language QA data, and the baseline models at GitHub.com/LLVM-AD/MAPLM.Keywords
This publication has 26 references indexed in Scilit:
- High‐definition map automatic annotation system based on active learningAI Magazine, 2023
- CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data2023
- WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models2023
- Road object detection for HD map: Full-element survey, analysis and perspectivesISPRS Journal of Photogrammetry and Remote Sensing, 2023
- ScanQA: 3D Question Answering for Spatial Scene Understanding2022
- SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events2021
- How to Keep HD Maps for Automated Driving Up To Date2020
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Autonomous Driving in the iCity—HD Maps as a Key Challenge of the Automotive IndustryEngineering, 2016
- VQA: Visual Question Answering2015