Holistic Autonomous Driving Understanding by Bird'View Injected Multi-Modal Large Models
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13668-13677
Abstract
The rise of multimodal large language models (MLLMs) has spurred interest in language-based driving tasks. However, existing research typically focuses on limited tasks and often omits key multi-view and temporal information which is crucial for robust autonomous driving. To bridge these gaps, we introduce NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17 subtasks, where each task demands holistic information (e.g., temporal, multi-view, and spatial), significantly elevating the challenge level. To obtain NuInstruct, we propose a novel SQL-based method to generate instruction-response pairs automatically, which is inspired by the driving logical progression of humans. We further present BEV-InMLLM, an end-to-end method for efficiently deriving instruction-aware Bird's-Eye-View (BEV) features, language-aligned for large language models. BEV-InMLLM integrates multi-view, spatial awareness, and temporal semantics to enhance MLLMs' capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g. 9% improvement on various tasks. We release our NuInstruct at https://github.com/xmed-lab/NuInstruct.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62306254)
- HKUST (HCIC-004)
This publication has 16 references indexed in Scilit:
- Referring Multi-Object Tracking2023
- Planning-oriented Autonomous Driving2023
- BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation2023
- DRAMA: Joint Risk Localization and Captioning in Driving2023
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- Textual Explanations for Self-Driving Vehicles2018
- Object Referring in Videos with Language and Human Gaze2018
- Deep Residual Learning for Image Recognition2016