Watermarking for Stable Diffusion Models
- 29 July 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 11 (21) , 35238-35249
Abstract
In the scenario of text data and image data interact of the Internet of Things (IoT) applications, the problem of copyright protection of the text-to-image models is threatened due to the replicability and portability of the neural network model. In order to solve this problem, we propose a model watermarking for the typical text-to-image diffusion models (DMs)–stable DMs (SDMs), which is a key aspect of the copyright protection of text-to-image models. Our scheme injects watermark into an SDM and makes the SDM generate watermark through a predefined prompt. The ownership of the SDM can be proved by the different output results of the model to the predefined prompt. The proposed method does not require raw training data and internal details of SDMs, which only need a predefined prompt and watermark to fine tune the pretrained SDM with minimal epoch. A large number of experiments show that our watermarking technology is effective, and can realize the copyright protection of the SDMs on the premise of less influence on the original function.Keywords
Funding Information
- Chenguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Commission (22CGA46)
- National Natural Science Foundation of China (62002214, 62302286, 62376148, U1936214, U22B2047)
This publication has 15 references indexed in Scilit:
- A Privacy-Preserving Hybrid Range Search Scheme Over Encrypted Electronic Medical Data in IoT SystemsIEEE Internet of Things Journal, 2023
- Application of QR Code Watermarking and Encryption in the Protection of Data Privacy of Intelligent Mouth-Opening TrainerIEEE Internet of Things Journal, 2023
- Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model2023
- A Data Integrity Authentication Scheme in WSNs Based on Double WatermarkIEEE Internet of Things Journal, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Vector Quantized Diffusion Model for Text-to-Image Synthesis2022
- Robust Watermarking for Deep Neural Networks via Bi-level Optimization2021
- NeuNAC: A novel fragile watermarking algorithm for integrity protection of neural networksInformation Sciences, 2021
- Watermarking Neural Networks With Watermarked ImagesIEEE Transactions on Circuits and Systems for Video Technology, 2020
- Embedding Watermarks into Deep Neural Networks2017