Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Simulation, and Real-Vehicle Experiment
Authors:
Preprint
Abstract
With the broader usage and highly successful development of Large Language Models (LLMs), there has been a growth of interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning ability, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to language interaction and decision-making. In this paper, we first introduce novel concepts and approaches to designing LLMs for autonomous driving (LLM4AD). Then, we propose a comprehensive benchmark for evaluating the instruction-following abilities of LLMs within the autonomous driving domain. Furthermore, we conduct a series of experiments on both simulation and real-world vehicle platforms, thoroughly evaluating the performance and potential of our LLM4AD systems. Our research highlights the significant potential of LLMs to enhance various aspects of autonomous vehicle technology, from perception and scene understanding to language interaction and decision-making.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- LingoQA: Visual Question Answering for Autonomous Driving2024
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models2024
- Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Code as Policies: Language Model Programs for Embodied Control2023
- End-to-End Urban Driving by Imitating a Reinforcement Learning Coach2021
- Accurate Path Tracking by Adjusting Look-Ahead Point in Pure Pursuit MethodInternational Journal of Automotive Technology, 2021
- An Open Approach to Autonomous VehiclesIEEE Micro, 2015
- Scan registration for autonomous mining vehicles using 3D‐NDTJournal of Field Robotics, 2007
- Congested traffic states in empirical observations and microscopic simulationsPhysical Review E, 2000