Query-Based Knowledge Sharing for Open-Vocabulary Multi-Label Classification
Open Access
- 21 November 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 21 (12) , 1-22
Abstract
Identifying labels that are unseen during training, known as multi-label zero-shot learning, is a non-trivial task in computer vision. Recent studies have increasingly focused on utilizing vision-language pre-training (VLP) models to recognize unseen labels in an open-vocabulary manner. However, these approaches like knowledge distillation have offered only modest performance gains. The challenge of fully harnessing the potential of VLP models for effective multi-label zero-shot learning remains open. In this work, an advanced query-based knowledge sharing framework is proposed to explore the multi-modal knowledge from VLP models for open-vocabulary multi-label classification. Specifically, we introduce a set of label-agnostic query tokens that are designed to capture essential and informative visual knowledge from input images. These tokens are subsequently shared across all labels, allowing them to select pertinent one as visual clues for accurate recognition. Then, by integrating the pre-trained knowledge of VLP models, these query tokens, trained on seen labels, can be efficiently generalized to the recognition of unseen labels. Additionally, we reformulate ranking learning into a form of classification to enable the magnitude of feature vectors for prediction, which significantly benefits label recognition. Experiment results show that our framework outperforms state-of-the-art methods in multi-label zero-shot learning task by a significant margin, reaching 4.2% and 2.4% in mAP on the NUS-WIDE and Open Images datasets, respectively. Code and models are available at https://github.com/jasonseu/QKS .Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62472092, 62172089, and 62106045)
- Natural Science Foundation of Jiangsu Province (BK20241751)
- Jiangsu Provincial Key Laboratory of Computer Networking Technology, Jiangsu Provincial Key Laboratory of Network and Information Security (BM2003201)
- Key Laboratory of Computer Network and Information Integration of Ministry of Education of China (93K-9)
This publication has 45 references indexed in Scilit:
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Open-Vocabulary DETR with Conditional Matching2022
- Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning2020
- The Open Images Dataset V4International Journal of Computer Vision, 2020
- Learning a Deep Embedding Model for Zero-Shot Learning2017
- Learning from Noisy Large-Scale Datasets with Minimal Supervision2017
- Zero-Shot Learning — The Good, the Bad and the Ugly2017
- CNN-RNN: A Unified Framework for Multi-label Image Classification2016