Abstract
Concept erasure has emerged as a practical safeguard for suppressing unsafe or unwanted concepts in text-to-image diffusion models. However, existing methods are primarily designed for text-triggered generation and are usually evaluated under text-only prompting. In this paper, we identify a practical robustness gap by showing that concepts suppressed under text-only evaluation can reappear after a concept-erased backbone is composed with a compatible structural controller and supplied with target-consistent, information-bearing structural conditions. We propose Structure-Guided Concept Reappearance (SGCR), an inference-time attack that combines a benign prompt with a structural condition, such as a Canny edge map, depth map, or segmentation map, to induce target-related outputs without retraining, gradient-based optimization, or modification of the defended backbone. Extensive experiments on explicit content, copyrighted artistic styles, and generic objects demonstrate that SGCR remains effective across multiple representative erasure defenses and structural modalities. These findings show that suppression established under text-only evaluation may not persist in controller-augmented diffusion pipelines. The demonstrated threat applies to local modular systems and services exposing compatible structural-control interfaces, rather than closed text-only APIs.
Funding Information
  • National Natural Science Foundation of China (62506339)
  • National Natural Science Foundation of China (62476250)
  • National Natural Science Foundation of China (62472335)
  • National Natural Science Foundation of China (62372137)
  • Zhejiang Provincial Natural Science Foundation of China (LQN25F010018)
  • Science and Technology Plan of Zhejiang Province (2025C01SA100283)
  • Specialized Foundation of Zhejiang University of Science and Technology (2025QN060)
  • Postgraduate Research and Innovation Foundation of Zhejiang University of Science and Technology (2024yjskc19)