ProSpect: Expanded Conditioning for the Personalization of Attribute-aware Image Generation
Authors:
- Yuxin ZhangYZYuxin Zhang
- Weiming DongWDWeiming Dong
- Fan TangFTFan Tang
- Nisha HuangNHNisha Huang
- Haibin HuangHHHaibin Huang
- Chongyang MaCMChongyang Ma
- Tong-Yee LeeTLTong-Yee Lee
- Oliver DeussenODOliver Deussen
- Changsheng XuCXChangsheng Xu
Preprint
Abstract
Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes like material, style, layout, etc. remains a challenge, leading to a lack of disentanglement and editability. To address this, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low- to high-frequency information, providing a new perspective on representing, generating, and editing images. We develop Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer stronger disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image/text-guided material/style/layout transfer/editing, achieving previously unattainable results with a single image input without fine-tuning the diffusion models.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- DiffStyler: Controllable Dual Diffusion for Text-Driven Image StylizationIEEE Transactions on Neural Networks and Learning Systems, 2024
- Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer2023
- Distributed data source verification in wireless sensor networksInformation Fusion, 2009
- Portalis2008
- Deciding equivalences among conjunctive aggregate queriesJournal of the ACM, 2007
- The patent holder's dilemmaCommunications of the ACM, 2007
- Interview with Bill KinderComputers in Entertainment, 2005
- Statecharts in use: structured analysis and object-orientation1998
- Vertex Types in Book-EmbeddingsSIAM Journal on Discrete Mathematics, 1989
- First-Order Dynamic Logic1979