XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition
- 1 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (10) , 1-18
Abstract
Inspired by the impressive success of image-text foundation models, recent works have proposed to adapt these foundation models to video data, leading to efficient and effective video models for open-vocabulary action recognition. However, through a comprehensive evaluation, our work finds that state-of the-art open-vocabulary action recognition models still struggle with generalization to video domains that they have not en countered. To address this limitation, we introduce generalizable open-vocabulary action recognition, which aims to develop action recognition models capable of generalizing to both novel action categories and unseen video domains. Our work contributes a novel model named XOV-Action to overcome two critical challenges: (1) understanding novel action concepts of open-set categories, and (2) mitigating the scenario discrepancy between training and test datasets. Specifically, XOV-Action first proposes to capture diverse action-related concepts by learning diversified elaboration representations, which enables better generalization to open-set action categories. Second, XOV-Action learns scene agnostic video representations to overcome the scene bias, which improves the generalization in unseen video domains. Addition ally, to evaluate models in generalizable open-vocabulary action recognition, we contribute a new cross-domain action benchmark named XOVABench, which covers multiple video domains with varying degrees of gaps and consists of both closed-set and open-set action categories. Extensive quantitative and qualitative experiments demonstrate that our proposed XOV-Action can effectively improve the action recognition performance for both closed-set and open-set categories across video domains.Keywords
Funding Information
- New Generation Artificial Intelligence-National Science and Technology (2025ZD0123100)
- NSFC (92470202)
- National Key Research and Development Program of China (2023YFA1008503)
- Guangdong NSF Project (2023B1515040025)
- Guangdong Key Research and Development Program (2024B0101040004)
This publication has 101 references indexed in Scilit:
- EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding2024
- InternVideo2: Scaling Foundation Models for Multimodal Video Understanding2024
- Leveraging Temporal Contextualization for Video Action Recognition2024
- Frozen CLIP Models are Efficient Video Learners2022
- Prompting Visual-Language Models for Efficient Video Understanding2022
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- Source-Free Video Domain Adaptation by Learning Temporal Consistency for Action Recognition2022
- Shuffle and Attend: Video Domain Adaptation2020
- RESOUND: Towards Action Recognition Without Representation Bias2018
- Temporal Relational Reasoning in Videos2018