Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
Preprint
Abstract
Data poisoning attacks manipulate training data to introduce unexpected behaviors into machine learning models at training time. For text-to-image generative models with massive training datasets, current understanding of poisoning attacks suggests that a successful attack would require injecting millions of poison samples into their training pipeline. In this paper, we show that poisoning attacks can be successful on generative models. We observe that training data per concept can be quite limited in these models, making them vulnerable to prompt-specific poisoning attacks, which target a model's ability to respond to individual prompts. We introduce Nightshade, an optimized prompt-specific poisoning attack where poison samples look visually identical to benign images with matching text prompts. Nightshade poison samples are also optimized for potency and can corrupt an Stable Diffusion SDXL prompt in <100 poison samples. Nightshade poison effects "bleed through" to related concepts, and multiple attacks can composed together in a single prompt. Surprisingly, we show that a moderate number of Nightshade attacks can destabilize general features in a text-to-image generative model, effectively disabling its ability to generate meaningful images. Finally, we propose the use of Nightshade` and similar tools as a last defense for content creators against web scrapers that ignore opt-out/do-not-crawl directives, and discuss possible implications for model trainers and content creators.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Pre-trained Encoders in Self-Supervised Learning Improve Secure and Privacy-preserving Supervised Learning2024
- Poisoning Web-Scale Training Datasets is Practical2024
- You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content2024
- Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning2023
- Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and DefensesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Bullseye Polytope: A Scalable Clean-Label Poisoning Attack with Improved Transferability2021
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts2021
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning2018
- Show and tell: A neural image caption generator2015