Open Access
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2251-2261
Abstract
We present Diffusion in Style, a simple method to adapt Stable Diffusion to any desired style, using only a small set of target images. It is based on the key observation that the style of the images generated by Stable Diffusion is tied to the initial latent tensor. Not adapting this initial latent tensor to the style makes fine-tuning slow, expensive, and impractical, especially when only a few target style images are available. In contrast, fine-tuning is much easier if this initial latent tensor is also adapted. Our Diffusion in Style is orders of magnitude more sample-efficient and faster. It also generates more pleasing images than existing approaches, as shown qualitatively and with quantitative comparisons.Keywords
This publication has 13 references indexed in Scilit:
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation2023
- ArtFID: Quantitative Evaluation of Neural Style Transfer2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Learning to generate line drawings that convey geometry and semantics2022
- Generative adversarial networksCommunications of the ACM, 2020
- Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Rethinking the Inception Architecture for Computer Vision2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015