Similar Modality Enhancement and Action Consistency Learning for Weakly Supervised Temporal Action Localization
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (5) , 4842-4850
Abstract
Weakly-supervised temporal action localization (WTAL) aims to identify and localize action instances in untrimmed videos using only video-level labels. Existing methods typically rely on original features from frozen pre-trained encoders designed for trimmed action classification (TAC) tasks, which inevitably introduces task discrepancy. Additionally, these methods often overlook the importance of considering action consistency from multiple perspectives, specifically the consistency in action processes and action semantics, both of which are crucial for the model's understanding of actions. To address these issues, we propose a novel WTAL method based on similar modality enhancement and action consistency learning (SEAL). First, we construct global descriptors for each action category, and use the pseudo-labels generated based on these descriptors to guide the model in learning more consistent representations, thereby mitigating task discrepancy. Second, we design two types of losses to achieve action consistency learning: process consistency loss, which penalizes candidate proposals that deviate from the action center to ensure the completeness of the action process, and semantic consistency loss, which employs local descriptors to help proposals of the same action category (especially those with apparent semantic confusion) learn similar feature distributions. Extensive experiments on the THUMOS14 and ActivityNet datasets demonstrate the superior performance of the proposed method compared to state-of-the-art methods.Keywords
This publication has 10 references indexed in Scilit:
- Temporal Action Localization in the Deep Learning Era: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Semantic and Temporal Contextual Correlation Learning for Weakly-Supervised Temporal Action LocalizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- Dual-Evidential Learning for Weakly-supervised Temporal Action Localization2022
- ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization2022
- Graph Convolutional Module for Temporal Action Localization in VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Revisiting Anchor Mechanisms for Temporal Action LocalizationIEEE Transactions on Image Processing, 2020
- Vision‐based garbage dumping action detection for real‐world surveillance platformETRI Journal, 2019
- Summarization of User-Generated Sports Video by Using Deep Action Recognition FeaturesIEEE Transactions on Multimedia, 2018
- Detecting Biological Motion for Human–Robot Interaction: A Link between Perception and ActionFrontiers in Robotics and AI, 2017
- The THUMOS challenge on action recognition for videos “in the wild”Computer Vision and Image Understanding, 2017