PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning
Top Cited Papers
- 12 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. PP (23798920) , 1-14
Abstract
Vehicle motion planning is an essential component of autonomous driving technology. Current rule-based vehicle motion planning methods perform satisfactorily in common scenarios but struggle to generalize to long-tailed situations. Meanwhile, learning-based methods have yet to achieve superior performance over rule-based approaches in large-scale closed-loop scenarios. To address these issues, we propose PlanAgent, the first mid-to-mid planning system based on a multi-modal large language model (MLLM). MLLM is used as a cognitive agent to introduce human-like knowledge, interpretability, and commonsense reasoning into closed-loop planning. Specifically, PlanAgent leverages the power of MLLM through three core modules. First, an Environment Transformation module constructs a Bird’s Eye View (BEV) map and a lane-graph-based textual description from the environment as inputs. Second, a Reasoning Engine module introduces a hierarchical chain-of-thought from scene understanding to lateral and longitudinal motion instructions, culminating in planner code generation. Last, a Reflection module is integrated to simulate and evaluate the generated planner for reducing MLLM’s uncertainty. PlanAgent is endowed with the common-sense reasoning and generalization capability of MLLM, which empowers it to effectively tackle both common and complex long-tailed scenarios. Our proposed PlanAgent is evaluated on the large-scale and challenging nuPlan benchmarks. A comprehensive set of experiments convincingly demonstrates that PlanAgent outperforms the existing state-of-the-art in the closed-loop motion planning task.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2022YFA1004000)
- Beijing Natural Science Foundation (4242052)
This publication has 47 references indexed in Scilit:
- GenAD: Generative End-to-End Autonomous Driving2024
- TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes2024
- VLP: Vision Language Planning for Autonomous Driving2024
- Enhancing Daily Life Through an Interactive Desktop Robotics System2024
- Planning-oriented Autonomous Driving2023
- End-to-End Object Detection with Transformers2020
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- Odin: Team VictorTango's entry in the DARPA Urban ChallengeJournal of Field Robotics, 2008
- Stanley: The robot that won the DARPA Grand ChallengeJournal of Field Robotics, 2006
- Congested traffic states in empirical observations and microscopic simulationsPhysical Review E, 2000