Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 24462-24471
Abstract
Recent text-to-image (T2I) diffusion models have revolutionized image editing by empowering users to control out-comes using natural language. However, the ease of image manipulation has raised ethical concerns, with the poten-tial for malicious use in generating deceptive or harmful content. To address the concerns, we propose an image im-munization approach named semantic attack to protect our images from being manipulated by malicious agents using diffusion models. Our approach focuses on disrupting the semantic understanding of T2I diffusion models regarding specific content. By attacking the cross-attention mecha-nism that encodes image features with text messages during editing, we distract the model's attention regarding the con-tent of our concern. Our semantic attack renders the model uncertain about the areas to edit, resulting in poorly edited images and contradicting the malicious editing attempts. In addition, by shifting the attack target towards intermediate attention maps from the final generated image, our approach substantially diminishes computational burden and alleviates GPU memory constraints in comparison to pre-vious methods. Moreover, we introduce timestep universal gradient updating to create timestep-agnostic perturbations effective across different input noise levels. By treating the full diffusion process as discrete denoising timesteps during the attack, we achieve equivalent or even superior immu-nization efficacy with nearly half the memory consumption of the previous method. Our contributions include a prac-tical and effective approach to safeguard images against malicious editing, and the proposed method offers robust immunization against various image inpainting and editing approaches, showcasing its potential for real-world appli-cations.Keywords
This publication has 32 references indexed in Scilit:
- Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models2023
- Null-text Inversion for Editing Real Images using Guided Diffusion Models2023
- Inversion-based Style Transfer with Diffusion Models2023
- Imagic: Text-Based Real Image Editing with Diffusion Models2023
- Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation2023
- SINE: SINgle Image Editing with Text-to-Image Diffusion Models2023
- TAFIM: Targeted Adversarial Attacks Against Facial Image Manipulations2022
- Blended Diffusion for Text-driven Editing of Natural Images2022
- Disrupting Deepfakes: Adversarial Attacks Against Conditional Image Translation Networks and Facial Manipulation Systems2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018