NetRoller: Interfacing General and Specialized Models for End-to-End Autonomous Driving
- 14 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Vehicular Technology
- Vol. 75 (5) , 7469-7482
Abstract
Integrating General Models (GMs) such as Large Language Models (LLMs), with Specialized Models (SMs) in autonomous driving tasks presents a promising approach to mitigating challenges in data diversity and model capacity of existing specialized driving models. However, this integration leads to problems of asynchronous systems, which arise from the distinct characteristics inherent in GMs and SMs. To tackle this challenge, we propose NetRoller, an adapter that incorporates a set of novel mechanisms to facilitate the seamless integration of GMs and specialized driving models. Specifically, our mechanisms for interfacing the asynchronous GMs and SMs are organized into three key stages. NetRoller first harvests semantically rich and computationally efficient representations from the reasoning processes of LLMs using an early stopping mechanism, which preserves critical insights on driving context while maintaining low overhead. It then applies learnable query embeddings, nonsensical embeddings, and positional layer embeddings to facilitate robust and efficient cross-modality translation. At last, it employs computationally efficient Query Shift and Feature Shift mechanisms to enhance the performance of SMs through few-epoch fine-tuning. Based on the mechanisms formalized in these three stages, NetRoller enables specialized driving models to operate at their native frequencies while maintaining situational awareness of the GM. Experiments conducted on the nuScenes dataset demonstrate that integrating GM through NetRoller significantly improves human similarity and safety in planning tasks, and it also achieves noticeable precision improvements in detection and mapping tasks for end-to-end autonomous driving. The code and models are available at https://github.com/Rex-sys-hk/NetRoller.Keywords
All Related Versions
Funding Information
- Guangzhou Education Bureau Project (2024312095)
This publication has 26 references indexed in Scilit:
- DriveLM: Driving with Graph Visual Question Answering2024
- Asynchronous Large Language Model Enhanced Planner for Autonomous Driving2024
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models2024
- Planning-oriented Autonomous Driving2023
- ST-P3: End-to-End Vision-Based Autonomous Driving via Spatial-Temporal Feature Learning2022
- Vision Transformer with Deformable Attention2022
- MP3: A Unified Model to Map, Perceive, Predict and Plan2021
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- End-To-End Interpretable Neural Motion Planner2019
- Commonsense Reasoning2018