Abstract
Accurately anticipating traffic accidents is a fundamental task for the safe and effective deployment of autonomous vehicles (AVs). However, existing models primarily rely on dashcam footage and often fail to generalize across varied driving scenarios due to their dependence on visual data and the rarity of high-risk events in datasets. These limitations undermine their robustness and reduce practical applicability in dynamic, unpredictable environments. To address these challenges, this study proposes a novel approach, termed MLTA, which integrates multimodal learning with the hypergraph attention network to hierarchically extract and capture cross-modal interaction. It leverages LLava-next, a multimodal large language model (MLLM) guided by the Chain-of-Thought (CoT) prompting paradigm, to produce context-aware interpretations of traffic scenes. This is further enhanced by a human-inspired attention mechanism that mimics the decision-making priorities of experienced human drivers. This combination enables more accurate identification of critical elements in a scene, improving both prediction precision and timeliness. Extensive experiments on four real-world datasets—DAD, A3D, CCD, and DADA-2000—show that our approach consistently outperforms state-of-the-art (SOTA) methods, demonstrating strong adaptability and robustness in complex driving environments.
Funding Information
  • Science and Technology Development Fund of Macau (0122/2024/RIB2, 0215/2024/AGJ, 001/2024/SKL)
  • Research Services and Knowledge Transfer Office, University of Macau (SRG2023-00037-IOTSC, MYRG-GRG2024-00284-IOTSC)
  • Shenzhen-Hong Kong-Macau Science and Technology Program Category C (SGDX20230821095159012)
  • State Key Laboratory of Intelligent Transportation System (2024-B001)
  • Jiangsu Provincial Science and Technology Program (BZ2024055)