IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion Models
Open Access
- 13 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Abstract
Reasoning about the intrinsic properties of an image, such as albedo, illumination, and surface geometry, is a long-standing problem with many applications in image editing and compositing. Existing solutions to this ill-posed problem either heavily rely on manually designed priors or learn priors from limited datasets that lack diversity. Hence, they fall short in generalizing to in-the-wild test scenarios. In this paper, we show that a large-scale text-to-image generation model trained on a massive amount of visual data can implicitly learn intrinsic image priors. In particular, we introduce a novel conditioning mechanism built on top of a pre-trained foundational image generation model to jointly predict multiple intrinsic modalities from an input image. We demonstrate that predicting different modalities in a collaborative manner improves the overall quality. This design also enables mixing datasets with annotations of only a subset of the modalities during training, contributing to the generalizability of our approach. Our method achieves state-of-the-art performance in intrinsic image decomposition, both qualitatively and quantitatively. We also demonstrate downstream image editing applications, such as relighting and retexturing.Keywords
Funding Information
- EPSRC CAMERA 2.0 (EP/T022523/1)
- UKRI MyWorld Strength in Places Programme (SIPF00006/1)
This publication has 56 references indexed in Scilit:
- Discriminative feature encoding for intrinsic image decompositionComputational Visual Media, 2023
- A Survey on Intrinsic Images: Delving Deep into Lambert and BeyondInternational Journal of Computer Vision, 2022
- Joint Learning of Intrinsic Images and Semantic Segmentation2018
- Shading Annotations in the Wild2017
- Unified Depth Prediction and Intrinsic Image Decomposition from a Single Image via Joint Convolutional Neural Fields2016
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Learning lightness from human judgement on relative reflectance2015
- Indoor Segmentation and Support Inference from RGBD Images2012
- Intrinsic images using optimization2011