Empowering Autonomous Driving with Large Language Models: A Safety Perspective
Preprint
Abstract
Autonomous Driving (AD) faces crucial hurdles for commercial launch, notably in the form of diminished public trust and safety concerns from long-tail unforeseen driving scenarios. This predicament is due to the limitation of deep neural networks in AD software, which struggle with interpretability and exhibit poor generalization capabilities in out-of-distribution and uncertain scenarios. To this end, this paper advocates for the integration of Large Language Models (LLMs) into the AD system, leveraging their robust common-sense knowledge, reasoning abilities, and human-interaction capabilities. The proposed approach deploys the LLM as an intelligent decision-maker in planning, incorporating safety verifiers for contextual safety learning to enhance overall AD performance and safety. We present results from two case studies that affirm the efficacy of our approach. We further discuss the potential integration of LLM for other AD software components including perception, prediction, and simulation. Despite the observed challenges in the case studies, the integration of LLMs is promising and beneficial for reinforcing both safety and performance in AD.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- DrivingDiffusion: Layout-Guided Multi-view Driving Scenarios Video Generation with Latent Diffusion Model2024
- Kinematics-aware Trajectory Generation and Prediction with Latent Stochastic Differential Modeling2024
- DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language ModelIEEE Robotics and Automation Letters, 2024
- TrafficGPT: Viewing, processing and interacting with traffic foundation modelsTransport Policy, 2024
- Safety-Assured Speculative Planning with Adaptive Prediction2023
- MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersections2023
- ADAPT: Action-aware Driving Caption Transformer2023
- Provably Safe Reinforcement Learning via Action Projection Using Reachability Analysis and Polynomial ZonotopesIEEE Open Journal of Control Systems, 2023
- ReachNN*: A Tool for Reachability Analysis of Neural-Network Controlled Systems2020
- Barrier certificates for nonlinear model validationAutomatica, 2006