Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization
Preprint
Abstract
As large-scale diffusion models continue to advance, they excel at producing high-quality images but often generate unwanted content, such as sexually explicit or violent content. Existing methods for concept removal generally guide the image generation process but can unintentionally modify unrelated regions, leading to inconsistencies with the original model. We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data. Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content. We evaluate our method on concept localization precision and replacement efficiency. Experimental results demonstrate that our method achieves superior precision in localizing target concepts and performs coherent concept replacement with minimal impact on non-target areas, outperforming existing approaches.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Diffusion Model is Secretly a Training-Free Open Vocabulary Semantic SegmenterIEEE Transactions on Image Processing, 2025
- Erasing Concepts from Diffusion Models2023
- SegGPT: Towards Segmenting Everything In Context2023
- Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion ModelsACM Transactions on Graphics, 2023