Multi-Modal GPT-4 Aided Action Planning and Reasoning for Self-driving Vehicles
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7325-7329
Abstract
Explainable decision-making is critical for building trust in autonomous vehicles. We investigate the use of a pre-trained large language model (LLM) to derive comprehensible driving decisions from multi-modal time-series data captured by a monocular camera on an autonomous vehicle. Leveraging a graph-of-thought structure, the LLM learns policies that perform robustly while generating natural language rationales. We generate a novel multi-modal dataset with sequential images, scene labels, and driving actions. Results demonstrate our method produces human- understandable explanations for its driving choices, providing transparency. Our work indicates incorporating language-based reasoning enables accountable and transparent decision-making for self-driving cars, making LLM a potential solution for autonomous driving.Keywords
This publication has 10 references indexed in Scilit:
- Graph of Thoughts: Solving Elaborate Problems with Large Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Drive Like a Human: Rethinking Autonomous Driving with Large Language Models2024
- InAction: Interpretable Action Decision Making for Autonomous Driving2022
- A Survey on Imitation Learning Techniques for End-to-End Autonomous VehiclesIEEE Transactions on Intelligent Transportation Systems, 2022
- OpenCDA: An Open Cooperative Driving Automation Framework Integrated with Co-Simulation2021
- Multi-Modal Fusion Transformer for End-to-End Autonomous Driving2021
- Explainable Object-Induced Action Decision for Autonomous Vehicles2020
- Grounding Human-To-Vehicle Advice for Self-Driving Vehicles2019
- Microscopic Traffic Simulation using SUMO2018
- Textual Explanations for Self-Driving Vehicles2018