Chain-of-Thought Guided Multimodal Large Language Models for Scene-Aware Accident Anticipation in Autonomous Driving
- 25 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 26 (11) , 19371-19380
Abstract
Accurately anticipating traffic accidents is a fundamental task for the safe and effective deployment of autonomous vehicles (AVs). However, existing models primarily rely on dashcam footage and often fail to generalize across varied driving scenarios due to their dependence on visual data and the rarity of high-risk events in datasets. These limitations undermine their robustness and reduce practical applicability in dynamic, unpredictable environments. To address these challenges, this study proposes a novel approach, termed MLTA, which integrates multimodal learning with the hypergraph attention network to hierarchically extract and capture cross-modal interaction. It leverages LLava-next, a multimodal large language model (MLLM) guided by the Chain-of-Thought (CoT) prompting paradigm, to produce context-aware interpretations of traffic scenes. This is further enhanced by a human-inspired attention mechanism that mimics the decision-making priorities of experienced human drivers. This combination enables more accurate identification of critical elements in a scene, improving both prediction precision and timeliness. Extensive experiments on four real-world datasets—DAD, A3D, CCD, and DADA-2000—show that our approach consistently outperforms state-of-the-art (SOTA) methods, demonstrating strong adaptability and robustness in complex driving environments.Keywords
Funding Information
- Science and Technology Development Fund of Macau (0122/2024/RIB2, 0215/2024/AGJ, 001/2024/SKL)
- Research Services and Knowledge Transfer Office, University of Macau (SRG2023-00037-IOTSC, MYRG-GRG2024-00284-IOTSC)
- Shenzhen-Hong Kong-Macau Science and Technology Program Category C (SGDX20230821095159012)
- State Key Laboratory of Intelligent Transportation System (2024-B001)
- Jiangsu Provincial Science and Technology Program (BZ2024055)
This publication has 39 references indexed in Scilit:
- LATTE: A Real-time Lightweight Attention-based Traffic Accident Anticipation EngineInformation Fusion, 2025
- Real-time accident anticipation for autonomous driving through monocular depth-enhanced 3D modelingAccident Analysis & Prevention, 2024
- A matched case-control analysis of autonomous vs human-driven vehicle accidentsNature Communications, 2024
- GPT-4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language modelsCommunications in Transportation Research, 2024
- Dynamic attention augmented graph network for video accident anticipationPattern Recognition, 2023
- THAT-Net: Two-layer hidden state aggregation based two-stream network for traffic accident predictionInformation Sciences, 2023
- Predicting Driver Attention in Critical Situations2019
- Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation2018
- Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization2017
- Anticipating Accidents in Dashcam Videos2017