Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selection
Open Access
- 19 February 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 21 (3) , 1-28
Abstract
The aim of multi-label few-shot image classification (ML-FSIC) is to assign semantic labels to images, in settings where only a small number of training examples are available for each label. A key feature of the multi-label setting is that an image often has several labels, which typically refer to objects appearing in different regions of the image. When estimating label prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data and the noisy nature of local features make this highly challenging. As a solution, we propose a strategy in which label prototypes are gradually refined. First, we initialize the prototypes using word embeddings, which allows us to leverage prior knowledge about the meaning of the labels. Second, taking advantage of these initial prototypes, we then use a Loss Change Measurement (LCM) strategy to select the local features from the training images (i.e., the support set) that are most likely to be representative of a given label. Third, we construct the final prototype of the label by aggregating these representative local features using a multi-modal cross-interaction mechanism, which again relies on the initial word embedding-based prototypes. Experiments on COCO, PASCAL VOC, NUS-WIDE, and iMaterialist show that our model substantially improves the current state-of-the-art.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- FSCIL-EACA: Few-Shot Class-Incremental Learning Network Based on Embedding Augmentation and Classifier Adaptation for Image ClassificationChinese Journal of Electronics, 2024
- Multi-label image classification with multi-layered multi-perspective dynamic semantic representationMachine Learning, 2023
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Few-shot partial multi-label learning via prototype rectificationKnowledge and Information Systems, 2023
- Cross Modal Adaptive Few-Shot Learning Based on Task DependenceChinese Journal of Electronics, 2023
- Feature learning network with transformer for multi-label image classificationPattern Recognition, 2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Microsoft COCO: Common Objects in Context2014
- Multi-Label ClassificationInternational Journal of Data Warehousing and Mining, 2007