StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7643-7655
Abstract
Content and style (C-S) disentanglement is a fundamental problem and critical challenge of style transfer. Existing approaches based on explicit definitions (e.g., Gram matrix) or implicit learning (e.g., GANs) are neither interpretable nor easy to control, resulting in entangled representations and less satisfying results. In this paper, we propose a new C-S disentangled framework for style transfer without using previous assumptions. The key insight is to explicitly extract the content information and implicitly learn the complementary style information, yielding interpretable and controllable C-S disentanglement and style transfer. A simple yet effective CLIP-based style disentanglement loss coordinated with a style reconstruction prior is introduced to disentangle C-S in the CLIP image space. By further leveraging the powerful style removal and generative ability of diffusion models, our framework achieves superior results than state of the art and flexible C-S disentanglement and trade-off control. Our work provides new insights into the C-S disentanglement in style transfer and demonstrates the potential of diffusion models for learning well-disentangled C-S characteristics.Keywords
All Related Versions
This publication has 61 references indexed in Scilit:
- CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer2022
- Evaluate and improve the quality of neural style transferComputer Vision and Image Understanding, 2021
- GLStyleNet: exquisite style transfer combining global and local pyramid featuresIET Computer Vision, 2020
- Multimodal Unsupervised Image-to-Image Translation2018
- A Style-Aware Content Loss for Real-Time HD Style Transfer2018
- Diverse Image-to-Image Translation via Disentangled Representations2018
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Statistics, vision, and the analysis of artistic styleWIREs Computational Statistics, 2011