AppleVLM: End-to-End Autonomous Driving With Advanced Perception and Planning-Enhanced Vision-Language Models
- 6 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 27 (6) , 7028-7043
Abstract
End-to-end autonomous driving has emerged as a promising paradigm integrating perception, decision-making, and control within a unified learning framework. Recently, Vision-Language Models (VLMs) have gained significant attention for their potential to enhance the robustness and generalization of end-to-end driving models in diverse and unseen scenarios. However, existing VLM-based approaches still face challenges, including suboptimal lane perception, language understanding biases, and difficulties in handling corner cases. To address these issues, we propose AppleVLM, an advanced perception and planning-enhanced VLM model for robust end-to-end driving. AppleVLM introduces a novel vision encoder and a planning strategy encoder to improve perception and decision-making. Firstly, the vision encoder fuses spatial-temporal information from multi-view images across multiple timesteps using a deformable transformer mechanism, enhancing robustness to camera variations and facilitating scalable deployment across different vehicle platforms. Secondly, unlike traditional VLM-based approaches, AppleVLM introduces a dedicated planning modality that encodes explicit Bird’s-Eye-View spatial information, mitigating language biases in navigation instructions. Finally, a VLM decoder fine-tuned by a hierarchical Chain-of-Thought integrates vision, language, and planning features to output robust driving waypoints. We evaluate AppleVLM in closed-loop experiments on two CARLA benchmarks, achieving state-of-the-art driving performance. Furthermore, we deploy AppleVLM on an AGV platform and successfully showcase real-world end-to-end autonomous driving in complex outdoor environments.Keywords
Funding Information
- National Natural Science Foundation of China (62573157)
- Science Center Program of National Natural Science Foundation of China (62188101)
- in part by Guangdong Provincial Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics (2023B1212010005)
- Chuzhou Science and Technology Program (SYSPG20241211173609005)
- Chuzhou Science and Technology Program (JCYJ20241202123714019)
This publication has 34 references indexed in Scilit:
- DriveLM: Driving with Graph Visual Question Answering2024
- End-to-End Autonomous Driving in CARLA: A SurveyIEEE Access, 2024
- VLP: Vision Language Planning for Autonomous Driving2024
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models2024
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding2024
- Autonomous driving system: A comprehensive surveyExpert Systems with Applications, 2023
- Hidden Biases of End-to-End Driving Models2023
- Planning-oriented Autonomous Driving2023
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- End-to-End Urban Driving by Imitating a Reinforcement Learning Coach2021