Understanding Implosion in Text-to-Image Generative Models
Open Access
- 2 December 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1211-1225
Abstract
Recent works show that text-to-image generative models are surprisingly vulnerable to a variety of poisoning attacks. Empirical results find that these models can be corrupted by altering associations between individual text prompts and associated visual features. Furthermore, a number of concurrent poisoning attacks can induce "model implosion," where the model becomes unable to produce meaningful images for unpoisoned prompts. These intriguing findings highlight the absence of an intuitive framework to understand poisoning attacks on these models. In this work, we establish the first analytical framework on robustness of image generative models to poisoning attacks, by modeling and analyzing the behavior of the cross-attention mechanism in latent diffusion models. We model cross-attention training as an abstract problem of "supervised graph alignment" and formally quantify the impact of training data by the hardness of alignment, measured by an Alignment Difficulty (AD) metric. The higher the AD, the harder the alignment. We prove that AD increases with the number of individual prompts (or concepts) poisoned. As AD grows, the alignment task becomes increasingly difficult, yielding highly distorted outcomes that frequently map meaningful text prompts to undefined or meaningless visual representations. As a result, the generative model implodes and outputs random, incoherent images at large. We validate our analytical framework through extensive experiments, and we confirm and explain the unexpected (and unexplained) effect of model implosion while producing new, unforeseen insights. Our work provides a useful tool for studying poisoning attacks against diffusion models and their defenses.Keywords
All Related Versions
Funding Information
- National Science Foundation (CNS-2241303, CNS-1949650)
This publication has 26 references indexed in Scilit:
- Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors2024
- Unified Concept Editing in Diffusion Models2024
- Editing Implicit Assumptions in Text-to-Image Diffusion Models2023
- Erasing Concepts from Diffusion Models2023
- TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets2023
- How to Backdoor Diffusion Models?2023
- Neural Distributed Image Compression with Cross-Attention Feature Alignment2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Analysis and Applications of Class-wise Robustness in Adversarial Training2021
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts2021