MobileCLIP2: Improving Multi-Modal Reinforced Training
Preprint
Abstract
Foundation image-text models such as CLIP with zero-shot capabilities enable a wide array of applications. MobileCLIP is a recent family of image-text models at 3-15ms latency and 50-150M parameters with state-of-the-art zero-shot accuracy. The main ingredients in MobileCLIP were its low-latency and light architectures and a novel multi-modal reinforced training that made knowledge distillation from multiple caption-generators and CLIP teachers efficient, scalable, and reproducible. In this paper, we improve the multi-modal reinforced training of MobileCLIP through: 1) better CLIP teacher ensembles trained on the DFN dataset, 2) improved captioner teachers trained on the DFN dataset and fine-tuned on a diverse selection of high-quality image-caption datasets. We discover new insights through ablations such as the importance of temperature tuning in contrastive knowledge distillation, the effectiveness of caption-generator fine-tuning for caption diversity, and the additive improvement from combining synthetic captions generated by multiple models. We train a new family of models called MobileCLIP2 and achieve state-of-the-art ImageNet-1k zero-shot accuracies at low latencies. In particular, we observe 2.2% improvement in ImageNet-1k accuracy for MobileCLIP2-B compared with MobileCLIP-B architecture. Notably, MobileCLIP2-S4 matches the zero-shot accuracy of SigLIP-SO400M/14 on ImageNet-1k while being 2$\times$ smaller and improves on DFN ViT-L/14 at 2.5$\times$ lower latency. We release our pretrained models (https://github.com/apple/ml-mobileclip) and the data generation code (https://github.com/apple/ml-mobileclip-dr). The data generation code makes it easy to create new reinforced datasets with arbitrary teachers using distributed scalable processing.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- FastVLM: Efficient Vision Encoding for Vision Language Models2025
- Active Data Curation Effectively Distills Large-Scale Multimodal Models2025
- EVA-02: A visual representation for neon genesisImage and Vision Computing, 2024
- InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions2023
- Masked Autoencoders Are Scalable Vision Learners2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Panoptic Feature Pyramid Networks2019
- Unified Perceptual Parsing for Scene Understanding2018
- Scene Parsing through ADE20K Dataset2017
- Deep visual-semantic alignments for generating image descriptions2015