Towards Reliable Verification of Unauthorized Data Usage in Personalized Text-to-Image Diffusion Models
Authors:
- 12 May 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2564-2582
Abstract
Text-to-image diffusion models are pushing the boundaries of what generative AI can achieve in our lives. Beyond their ability to generate general images, new personalization techniques have been proposed to customize the pretrained base models for crafting images with specific themes or styles. Such a lightweight solution, enabling AI practitioners and developers to easily build their own personalized models, also poses a new concern regarding whether the personalized models are trained from unauthorized data. A promising solution is to proactively enable data traceability in generative models, where data owners embed external coatings (e.g., image watermarks or backdoor triggers) onto the datasets before releasing. Later the models trained over such datasets will also learn the coatings and unconsciously reproduce them in the generated mimicries, which can be extracted and used as the data usage evidence. However, we identify the existing coatings cannot be effectively learned in personalization tasks, making the corresponding verification less reliable. In this paper, we introduce SIREN, a novel methodology to proactively trace unauthorized data usage in black-box personalized text-to-image diffusion models. Our approach optimizes the coating in a delicate way to be recognized by the model as a feature relevant to the personalization task, thus significantly improving its learnability. We also utilize a human perceptual-aware constraint, a hypersphere classification technique, and a hypothesis-testing-guided verification method to enhance the stealthiness and detection accuracy of the coating. The effectiveness of SIREN is verified through extensive experiments on a diverse set of benchmark datasets, models, and learning algorithms. SIREN is also effective in various real-world scenarios and evaluated against potential countermeasures. Our code is publicly available here.Keywords
All Related Versions
Funding Information
- National Research Foundation
- National Research Foundation, Singapore
- National Natural Science Foundation of China (62202340)
- Fundamental Research Funds for the Central Universities (2042023kf0121)
This publication has 30 references indexed in Scilit:
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Vector Quantized Diffusion Model for Text-to-Image Synthesis2022
- PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation2021
- Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules2020
- HiDDeN: Hiding Data With Deep Networks2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Deep neural networks are easily fooled: High confidence predictions for unrecognizable images2015
- The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observationsColor Research & Application, 2004
- The development of the CIE 2000 colour‐difference formula: CIEDE2000Color Research & Application, 2001