CPR: Retrieval Augmented Generation for Copyright Protection
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12374-12384
Abstract
Retrieval Augmented Generation (RAG) is emerging as a flexible and robust technique to adapt models to private users data without training, to handle credit attribution, and to allow efficient machine unlearning at scale. However, RAG techniques for image generation may lead to parts of the retrieved samples being copied in the model's output. To reduce risks of leaking private information contained in the retrieved set, we introduce Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models. CPR allows to condition the output of diffusion models on a set of retrieved images, while also guaranteeing that unique identifiable information about those example is not exposed in the generated outputs. In particular, it does so by sampling from a mixture of public (safe) distribution and private (user) distribution by merging their diffusion scores at inference. We prove that CPR satisfies Near Access Freeness (NAF) which bounds the amount of information an attacker may be able to extract from the generated images. We provide two algorithms for copyright protection, CPR-KL and CPR-Choose. Unlike previously proposed rejection-sampling-based NAF methods, our methods enable efficient copyright-protected sampling with a single run of backward diffusion. We show that our method can be applied to any pre-trained conditional diffusion model, such as Stable Diffusion or unCLIP. In particular, we empirically show that applying CPR on top of unCLIP improves quality and text-to-image alignment of the generated results (81.4 to 83.17 on TIFA benchmark), while enabling credit attribution, copy-right protection, and deterministic, constant time, unlearning.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices2023
- Visual Prompt Tuning2022
- Compositional Visual Generation with Composable Diffusion Models2022
- Mixed-Privacy Forgetting in Deep Networks2021
- The Legal Frameworks of the Right to Request the Deletion of Personal Data in the EU, the U.S. and Japan and the Right to Be Forgotten: A Study Focusing on Search Businesses2020
- Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks2020
- The Right to Be Forgotten in the Digital Age: The Challenges of Data Protection Beyond BordersGerman Law Journal, 2020
- Microsoft COCO: Common Objects in Context2014
- Differential Privacy2006
- Annealed importance samplingStatistics and Computing, 2001