Empowering Autonomous Driving with Large Language Models: A Safety Perspective
Preprint
Abstract
Autonomous Driving (AD) encounters significant safety hurdles in long-tail unforeseen driving scenarios, largely stemming from the non-interpretability and poor generalization of the deep neural networks within the AD system, particularly in out-of-distribution and uncertain data. To this end, this paper explores the integration of Large Language Models (LLMs) into AD systems, leveraging their robust common-sense knowledge and reasoning abilities. The proposed methodologies employ LLMs as intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning, for enhancing driving performance and safety. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine. Demonstrating superior performance and safety metrics compared to state-of-the-art approaches, our approach shows the promising potential for using LLMs for autonomous vehicles.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- DrivingDiffusion: Layout-Guided Multi-view Driving Scenarios Video Generation with Latent Diffusion Model2024
- Kinematics-aware Trajectory Generation and Prediction with Latent Stochastic Differential Modeling2024
- DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language ModelIEEE Robotics and Automation Letters, 2024
- TrafficGPT: Viewing, processing and interacting with traffic foundation modelsTransport Policy, 2024
- Safety-Assured Speculative Planning with Adaptive Prediction2023
- MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersections2023
- ADAPT: Action-aware Driving Caption Transformer2023
- Differentiable Safe Controller Design Through Control Barrier FunctionsIEEE Control Systems Letters, 2022
- ReachNN*: A Tool for Reachability Analysis of Neural-Network Controlled Systems2020
- Barrier certificates for nonlinear model validationAutomatica, 2006