Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Preprint
Abstract
Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. However, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the requirements, we introduce Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest2025
- GRiT: A Generative Region-to-Text Transformer for Object Understanding2024
- Improved Baselines with Visual Instruction Tuning2024
- InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning2024
- GLaMM: Pixel Grounding Large Multimodal Model2024
- CPT: Colorful Prompt Tuning for pre-trained vision-language modelsAI Open, 2024
- What does CLIP know about a red circle? Visual prompt engineering for VLMs2023
- ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation2023
- Region Driven Remote Sensing Image CaptioningProcedia Computer Science, 2019
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017