Raising the Bar of AI-generated Image Detection with CLIP
Authors:
Preprint
Abstract
The aim of this work is to explore the potential of pre-trained vision-language models (VLMs) for universal detection of AI-generated images. We develop a lightweight detection strategy based on CLIP features and study its performance in a wide variety of challenging scenarios. We find that, contrary to previous beliefs, it is neither necessary nor convenient to use a large domain-specific dataset for training. On the contrary, by using only a handful of example images from a single generative model, a CLIP-based detector exhibits surprising generalization ability and high robustness across different architectures, including recent commercial tools such as Dalle-3, Midjourney v5, and Firefly. We match the state-of-the-art (SoTA) on in-distribution data and significantly improve upon it in terms of generalization to out-of-distribution data (+6% AUC) and robustness to impaired/laundered data (+13%). Our project is available at https://grip-unina.github.io/ClipBased-SyntheticImageDetection/Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification2022
- Discovering Transferable Forensic Features for CNN-Generated Images Detection2022
- Detecting Generated Images by Real Images2022
- FingerprintNet: Synthesized Fingerprints for Generated Image Detection2022
- A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- What Makes Fake Images Detectable? Understanding Properties that Generalize2020
- Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions2020
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Microsoft COCO: Common Objects in Context2014