Suit the Remedy to the Case: Precise Backdoor Elimination for Text-to-Image Diffusion Models via Model Editing
Open Access
- 9 November 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 125-134
Abstract
With the widespread adoption of Text-to-Image (T2I) Diffusion Models (DMs) across various applications, their security has attracted increasing attention. In particular, these models are highly vulnerable to backdoor attacks, where adversaries implant hidden triggers during training to manipulate model outputs at inference time. To deal with the backdoor attacks, existing defense methods typically require substantial training data and employ coarse-grained alignment, leading to catastrophic forgetting and semantic degradation, respectively. To defend against backdoor attacks while preventing performance degradation, this paper first analyzes the behavior of poisoned models and finds that i) the neuron activation variation distributions of trigger tokens are significantly different from those of normal tokens, and ii) the embeddings of non-semantic trigger tokens are far from the manifold space constructed by all the normal tokens in the vocabulary. Based on the findings, we propose ManiEdit, a novel model-edit-based backdoor elimination framework for T2I DMs. Specifically, we propose an activation-based module localization methodology to pinpoint the backdoored module by calculating the Neuron Activation Variations (NAV), enabling precise parameter recoupling within restricted areas to safeguard benign parameters from collateral damage. To sever malicious associations and restore the original semantics of triggers, we present a manifold-driven trigger categorization strategy, which detects triggers via NAV within the localized modules, subsequently applying syntactic and geometric constraints to distinguish non-semantic from semantic triggers. By aligning non-semantic triggers with empty prompts and realigning semantic triggers with denoising-optimized semantic anchors, ManiEdit can achieve efficient elimination without expensive retraining. Comprehensive experiments demonstrate the effectiveness of ManiEdit in defending against various state-of-the-art backdoor attack methods.Keywords
Funding Information
- Natural Science Foundation of China (62272170)
This publication has 25 references indexed in Scilit:
- Efficient Input-Level Backdoor Defense on Text-to-Image Synthesis via Neuron Activation Variation2025
- Panorama Generation From NFoV Image Done Right2025
- UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models2025
- Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models2025
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation2025
- DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving2025
- CAT-DM: Controllable Accelerated Virtual Try-On with Diffusion Model2024
- Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- A New Backdoor Attack in CNNS by Training Set Corruption Without Label Poisoning2019