SuperCLIP: CLIP with Simple Classification Supervision
Authors:
Preprint
Abstract
Contrastive Language-Image Pretraining (CLIP) achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space. However, recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text, and this issue becomes even more pronounced when dealing with long and detailed captions. This stems from CLIP's training objective, which optimizes only global image-text similarity and overlooks token-level supervision - limiting its ability to achieve fine-grained visual-text alignment. To address this, we propose SuperCLIP, a simple yet effective framework that augments contrastive learning with classification-based supervision. By adding only a lightweight linear layer to the vision encoder, SuperCLIP leverages token-level cues to enhance visual-textual alignment - with just a 0.077% increase in total FLOPs, and no need for additional annotated data. Experiments show that SuperCLIP consistently improves zero-shot classification, image-text retrieval, and purely visual tasks. These gains hold regardless of whether the model is trained on original web data or rich re-captioned data, demonstrating SuperCLIP's ability to recover textual supervision in both cases. Furthermore, SuperCLIP alleviates CLIP's small-batch performance drop through classification-based supervision that avoids reliance on large batch sizes. Code and models will be made open source.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive TrainingIEEE Transactions on Image Processing, 2023
- Evaluating Object Hallucination in Large Vision-Language Models2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Grounded Language-Image Pre-training2022
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover ClassificationIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Remote Sensing Image Scene Classification: Benchmark and State of the ArtProceedings of the IEEE, 2017
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998