Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14066-14075
Abstract
Multiple clustering has gained significant attention in recent years due to its potential to reveal multiple hidden structures of data from different perspectives. The advent of deep multiple clustering techniques has notably advanced the performance by uncovering complex patterns and relationships within large datasets. However, a major challenge arises as users often do not need all the clusterings that algorithms generate, and figuring out the one needed requires a substantial understanding of each clustering result. Traditionally, aligning a user's brief keyword of interest with the corresponding vision components was challenging, but the emergence of multi-modal and large language models (LLMs) has begun to bridge this gap. In response, given unlabeled target visual data, we propose Multi-MaP, a novel method employing a multi-modal proxy learning process. It leverages CLIP encoders to extract coherent text and image embeddings, with GPT-4 integrating users' interests to formulate effective textual contexts. Moreover, reference word constraint and concept-level constraint are designed to learn the optimal text proxy according to the user's interest. Multi-MaP not only adeptly captures a user's interest via a keyword but also facilitates identifying relevant clusterings. Our extensive experiments show that Multi-MaP consistently outperforms state-of-the-art methods in all benchmark multi-clustering vision tasks. Our code is available at https://github.com/Alexander-Yao/Multi-MaP.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- AugDMC: Data Augmentation Guided Deep Multiple ClusteringProcedia Computer Science, 2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Subspace multi-clustering: a reviewKnowledge and Information Systems, 2017
- Non-redundant multiple clustering by nonnegative matrix factorizationMachine Learning, 2016
- Finding multiple stable clusteringsKnowledge and Information Systems, 2016
- WordNet2010
- The Amsterdam Library of Object ImagesInternational Journal of Computer Vision, 2005
- Pattern Recognition and Machine Learning1971