Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
Authors:
Top Cited Papers
- 19 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 807-825
Abstract
Trained on billions of images, diffusion-based text-to-image models seem impervious to traditional data poisoning attacks, which typically require poison samples approaching 20% of the training set. In this paper, we show that state-of-the-art text-to-image generative models are in fact highly vulnerable to poisoning attacks. Our work is driven by two key insights. First, while diffusion models are trained on billions of samples, the number of training samples associated with a specific concept or prompt is generally on the order of thousands. This suggests that these models will be vulnerable to prompt-specific poisoning attacks that corrupt a model’s ability to respond to specific targeted prompts. Second, poison samples can be carefully crafted to maximize poison potency to ensure success with very few samples.We introduce Nightshade, a prompt-specific poisoning attack optimized for potency that can completely control the output of a prompt in Stable Diffusion’s newest model (SDXL) with less than 100 poisoned training samples. Nightshade also generates stealthy poison images that look visually identical to their benign counterparts, and produces poison effects that "bleed through" to related concepts. More importantly, a moderate number of Nightshade attacks on independent prompts can destabilize a model and disable its ability to generate images for any and all prompts. Finally, we propose the use of Nightshade and similar tools as a defense for content owners against web scrapers that ignore opt-out/do-not-crawl directives, and discuss potential implications for both model trainers and content owners.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Backdoor Attacks Against Deep Learning Systems in the Physical World2021
- Understanding the Behaviour of Contrastive Loss2021
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts2021
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis2019
- Robust Physical-World Attacks on Deep Learning Visual Classification2018
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Show and tell: A neural image caption generator2015
- Microsoft COCO: Common Objects in Context2014