Leveraging Multimodal LLM Descriptions of Activity for Explainable Semi-Supervised Video Anomaly Detection
Preprint
Abstract
Existing semi-supervised video anomaly detection (VAD) methods often struggle with detecting complex anomalies involving object interactions and generally lack explainability. To overcome these limitations, we propose a novel VAD framework leveraging Multimodal Large Language Models (MLLMs). Unlike previous MLLM-based approaches that make direct anomaly judgments at the frame level, our method focuses on extracting and interpreting object activity and interactions over time. By querying an MLLM with visual inputs of object pairs at different moments, we generate textual descriptions of the activity and interactions from nominal videos. These textual descriptions serve as a high-level representation of the activity and interactions of objects in a video. They are used to detect anomalies during test time by comparing them to textual descriptions found in nominal training videos. Our approach inherently provides explainability and can be combined with many traditional VAD methods to further enhance their interpretability. Extensive experiments on benchmark datasets demonstrate that our method not only detects complex interaction-based anomalies effectively but also achieves state-of-the-art performance on datasets without interaction anomalies.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Deep Learning for Video Anomaly Detection: A ReviewIEEE Transactions on Neural Networks and Learning Systems, 2026
- Fast and Lightweight Vision-Language Model for Adversarial Traffic Sign DetectionElectronics, 2024
- A critical study on the recent deep learning based semi-supervised video anomaly detection methodsMultimedia Tools and Applications, 2023
- X-MAN: Explaining multiple sources of anomalies in video2021
- A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in VideoIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners2021
- A Survey of Single-Scene Video Anomaly DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Scene-Aware Context Reasoning for Unsupervised Abnormal Event Detection in Videos2020