A Survey on Multimodal Large Language Models for Autonomous Driving
Authors:
Top Cited Papers
- 1 January 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 958-979
Abstract
With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.Keywords
All Related Versions
This publication has 108 references indexed in Scilit:
- High‐definition map automatic annotation system based on active learningAI Magazine, 2023
- In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and BeyondInternational Journal of Computer Vision, 2023
- AI art in architectureAI in Civil Engineering, 2023
- Simple Open-Vocabulary Object Detection2022
- A survey of deep learning techniques for autonomous drivingJournal of Field Robotics, 2019
- A review on safety failures, security attacks, and available countermeasures for autonomous vehiclesAd Hoc Networks, 2019
- Textual Explanations for Self-Driving Vehicles2018
- Stanley: The robot that won the DARPA Grand ChallengeJournal of Field Robotics, 2006
- Blocks World revisitedArtificial Intelligence, 2001
- The Present Status of Automatic Translation of Languages1960