SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitution
Open Access
- 2 December 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1166-1180
Abstract
Advanced text-to-image models such as DALL⋅E 2, Midjourney, and Stable Diffusion can generate highly realistic images, raising significant concerns regarding the potential proliferation of unsafe content. This includes adult, violent, or deceptive imagery of political figures. Despite claims of rigorous safety mechanisms implemented in these models to restrict the generation of Not-Safe-For-Work (NSFW) content, we successfully devise and exhibit the first prompt attacks on Midjourney, producing abundant photorealistic NSFW images. We reveal the fundamental principles of such prompt attacks and strategically substitute high-risk sections within a suspect prompt to evade closed-source safety measures. Our novel framework, SurrogatePrompt, systematically generates attack prompts, utilizing large language models and image-to-text modules to automate attack prompt creation at scale. Evaluation results disclose an 88% success rate in bypassing Midjourney's proprietary safety filter with our attack prompts, leading to counterfeit images depicting political figures in violent scenarios with high probability. We also demonstrate attacks generating explicit adult-themed imagery. Both subjective and objective assessments validate that the images generated from our attack prompts present considerable safety hazards.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2023YFB2904000, 2023YFB2904001)
- National Natural Science Foundation of China (62172359,62072395,62122066)
- Zhejiang Provincial Natural Science Foundation of China under Grant (LD24F020014)
- Joint Funds of the National Natural Science Foundation of China (U20A20178)
- Hangzhou Leading Innovation and Entrepreneurship Team (TD2020003)
This publication has 6 references indexed in Scilit:
- Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models2023
- RIATIG: Reliable and Imperceptible Adversarial Text-to-Image Generation with Natural Prompts2023
- Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models2023
- Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- ArcFace: Additive Angular Margin Loss for Deep Face RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021