Espresso: Robust Concept Filtering in Text-to-Image Models
Preprint
Abstract
Diffusion-based text-to-image (T2I) models generate high-fidelity images for given textual prompts. They are trained on large datasets scraped from the Internet, potentially containing unacceptable concepts (e.g., copyright infringing or unsafe). Retraining T2I models after filtering out unacceptable concepts in the training data is inefficient and degrades utility. Hence, there is a need for concept removal techniques (CRTs) which are effective in removing unacceptable concepts, utility-preserving on acceptable concepts, and robust against evasion with adversarial prompts. None of the prior filtering and fine-tuning CRTs satisfy all these requirements simultaneously. We introduce Espresso, the first robust concept filter based on Contrastive Language-Image Pre-Training (CLIP). It identifies unacceptable concepts by projecting the generated image's embedding onto the vector connecting unacceptable and acceptable concepts in the joint text-image embedding space. This ensures robustness by restricting the adversary to adding noise only along this vector, in the direction of the acceptable concept. Further fine-tuning Espresso to separate embeddings of acceptable and unacceptable concepts, while preserving their pairing with image embeddings, ensures both effectiveness and utility. We evaluate Espresso on eleven concepts to show that it is effective (~5% CLIP accuracy on unacceptable concepts), utility-preserving (~93% normalized CLIP score on acceptable concepts), and robust (~4% CLIP accuracy on adversarial prompts for unacceptable concepts). Finally, we present theoretical bounds for the certified robustness of Espresso against adversarial prompts, and an empirical analysis.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitution2024
- Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers2024
- Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models2024
- All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Ablating Concepts in Text-to-Image Diffusion Models2023
- Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models2023
- StyleGAN-NADAACM Transactions on Graphics, 2022
- Enhancing Robustness of Classifiers Based on PCA2021
- Twenty Thousand Leagues under the Seas2019
- Enhancing robustness of machine learning systems via data transformations2018