ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
Top Cited Papers
Open Access
- 5 December 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 42 (6) , 1-14
Abstract
Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available at https://github.com/zyxElsa/ProSpect.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61832016, 62102162, U20B2070)
- Deutsche Forschungsgemeinschaft (413891298)
- Beijing Natural Science Foundation (L221013)
- National Key R&D Program of China (2020AAA0106200)
- National Science and Technology Council (111-2221-E-006-112-MY3)
This publication has 20 references indexed in Scilit:
- VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance2022
- Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning2022
- StyleGAN-NADAACM Transactions on Graphics, 2022
- Text to Image Generation with Semantic-Spatial Aware GAN2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- StyTr2: Image Style Transfer with Transformers2022
- CLIPstyler: Image Style Transfer with a Single Text Condition2022
- DRIT++: Diverse Image-to-Image Translation via Disentangled RepresentationsInternational Journal of Computer Vision, 2020
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018