DreamBooth++: Boosting Subject-Driven Generation via Region-Level References Packing
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 11013-11021
Abstract
DreamBooth has demonstrated significant potential in subject-driven text-to-image generation, especially in scenarios requiring precise preservation of a subject's appearance. However, it still suffers from inefficiency and requires extensive iterative training to customize concepts using a small set of reference images. To address these issues, we introduce DreamBooth++, a region-level training strategy designed to significantly improve the efficiency and effectiveness of learning specific subjects. In particular, our approach employs a region-level data re-formulation technique that packs a set of reference images into a single sample, significantly reducing computational costs. Moreover, we adapt convolution and self-attention layers to ensure their processings are restricted within individual regions. Thus their operational scope (i.e., receptive field) can be preserved within a single subject, avoiding generating multiple sub-images within a single image. Last but not least, we design a text-guided prior regularization between our model and the pretrained one to preserve the original semantic generation ability. Comprehensive experiments demonstrate that our training strategy not only accelerates the subject-learning process but also significantly boosts fidelity to both subject and prompts in subject-driven generation.Keywords
Funding Information
- Ministry of Science and Technology of the People's Republic of China (No. 2021ZD0111700)
This publication has 8 references indexed in Scilit:
- ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion ModelsACM Transactions on Graphics, 2023
- MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing2023
- MagicFusion: Boosting Text-to-Image Generation Performance by Fusing Diffusion Models2023
- ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Emerging Properties in Self-Supervised Vision Transformers2021
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018