Alpha-CLIP: A CLIP Model Focusing on Wherever you Want
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13019-13029
Abstract
Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. How-ever, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the require we introduce Alpha-CLIp, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks. Our project is with codes and models available is linked to https://aleafy.github.io/alpha-clip/.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Detecting Twenty-Thousand Classes Using Image-Level Supervision2022
- Exploiting Unlabeled Data with Vision and Language Models for Object Detection2022
- Extract Free Dense Labels from CLIP2022
- LVIS: A Dataset for Large Vocabulary Instance Segmentation2019
- Region Driven Remote Sensing Image CaptioningProcedia Computer Science, 2019
- MAttNet: Modular Attention Network for Referring Expression Comprehension2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Modeling Context in Referring Expressions2016
- Generation and Comprehension of Unambiguous Object Descriptions2016
- ImageNet: A large-scale hierarchical image database2009