Multilingual-Prompt-Guided Directional Feature Learning for Weakly Supervised Video Anomaly Detection
- 17 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (11) , 9994-10011
Abstract
Weakly supervised video anomaly detection has gained attention for its effective performance and cost-efficient annotation, using video-level labels to distinguish between normal and abnormal patterns. However, challenges arise from the diversity and incompleteness of anomalous events, complicating feature learning. Vision-language models offer promising approaches, but designing precise prompts remains difficult. This is because accommodating the diverse range of normal and anomalous scenarios in real-world settings is challenging, and the workload is significant. To tackle these issues, we propose integrating multilingualism and multiple prompts to improve feature learning. By utilizing prompts in various languages to define “anomaly” and “normalcy,” we tackle these concepts across different linguistic domains. In each domain, multiple prompts are employed for adaptive top-K prompt selection of snippets. To enhance visual feature learning, a multi-granularity attention module combining Transformer and Mamba is designed. Mamba’s long-range adaptation selection builds fine-grained temporal correlations among coarse-grained snippets, while Transformer enhances fine-grained information guided by coarse-grained information. Alongside a multilingual prompt guidance loss, we introduce a gradual directional loss to jointly optimize visual feature distribution and the top-K prompt selection. Our method demonstrates effectiveness on four video datasets and provides generalizability analyses on two medical datasets, including EMG and ECG temporal data.Keywords
Funding Information
- National Natural Science Foundation of China (62371219, 62271221)
- Basic and Applied Basic Research Foundation of Guangdong Province (2023A1515011260, 2024A1515140070)
- SERC Central Research Fund
This publication has 51 references indexed in Scilit:
- Multi-Scale Transformer Pyramid Networks for Multivariate Time Series ForecastingIEEE Access, 2024
- Prompting Visual-Language Models for Efficient Video Understanding2022
- MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection2021
- Not only Look, But Also Listen: Learning Multimodal Violence Detection Under Weak Supervision2020
- Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection2019
- Real-World Anomaly Detection in Surveillance Videos2018
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Dynamic Image Networks for Action Recognition2016
- The recognition of human movement using temporal templatesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- The impact of the MIT-BIH Arrhythmia DatabaseIEEE Engineering in Medicine and Biology Magazine, 2001