Large Language Models for Autonomous Driving: Real-World Experiments
Authors:
Preprint
Abstract
Autonomous driving systems are increasingly popular in today's technological landscape, where vehicles with partial automation have already been widely available on the market, and the full automation era with "driverless" capabilities is near the horizon. However, accurately understanding humans' commands, particularly for autonomous vehicles that have only passengers instead of drivers, and achieving a high level of personalization remain challenging tasks in the development of autonomous driving systems. In this paper, we introduce a Large Language Model (LLM)-based framework Talk-to-Drive (Talk2Drive) to process verbal commands from humans and make autonomous driving decisions with contextual information, satisfying their personalized preferences for safety, efficiency, and comfort. First, a speech recognition module is developed for Talk2Drive to interpret verbal inputs from humans to textual instructions, which are then sent to LLMs for reasoning. Then, appropriate commands for the Electrical Control Unit (ECU) are generated, achieving a 100% success rate in executing codes. Real-world experiments show that our framework can substantially reduce the takeover rate for a diverse range of drivers by up to 90.1%. To the best of our knowledge, Talk2Drive marks the first instance of employing an LLM-based system in a real-world autonomous driving environment.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs2024
- ADAPT: Action-aware Driving Caption Transformer2023
- Vehicle crash simulations for safety: Introduction of connected and automated vehicles on the roadwaysAccident Analysis & Prevention, 2023
- Standards for passenger comfort in automated vehicles: Acceleration and jerkApplied Ergonomics, 2022
- Relating individual motion sickness levels to subjective discomfort ratingsExperimental Brain Research, 2022
- Accurate Path Tracking by Adjusting Look-Ahead Point in Pure Pursuit MethodInternational Journal of Automotive Technology, 2021
- Exploring Personalised Autonomous Vehicles to Influence User TrustCognitive Computation, 2020
- A survey of deep learning techniques for autonomous drivingJournal of Field Robotics, 2019
- An experimental and theoretical investigation into simultaneous localisation and map building2008
- Scan registration for autonomous mining vehicles using 3D‐NDTJournal of Field Robotics, 2007