Controlling the Latent Diffusion Model for Generative Image Shadow Removal via Residual Generation
Open Access
- 3 October 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Abstract
Large-scale generative models have achieved remarkable advancements in various visual tasks, yet their application to shadow removal in images remains challenging. These models often generate diverse, realistic details without adequate focus on fidelity, failing to meet the crucial requirements of shadow removal, which necessitates precise preservation of image content. In contrast to prior approaches that aimed to regenerate shadow-free images from scratch, this paper utilizes diffusion models to generate and refine image residuals. This strategy fully uses the inherent detailed information within shadowed images, resulting in a more efficient and faithful reconstruction of shadow-free content. Additionally, to prevent the accumulation of errors during the generation process, a cross-timestep self-enhancement training strategy is proposed. This strategy leverages the network itself to augment the training data, not only increasing the volume of data but also enabling the network to dynamically correct its generation trajectory, ensuring a more accurate and robust output. In addition, to address the loss of original details in the process of image encoding and decoding of large generative models, a content-preserved encoder-decoder structure is designed with a control mechanism and multi-scale skip connections to achieve high-fidelity shadow-free image reconstruction. Experimental results demonstrate that the proposed method can reproduce high-quality results based on a large latent diffusion prior and faithfully preserve the original contents in shadow regions.Keywords
All Related Versions
This publication has 46 references indexed in Scilit:
- Style-Guided Shadow Removal2022
- Bijective Mapping Network for Shadow Removal2022
- Auto-Exposure Fusion for Single-Image Shadow Removal2021
- Taming Transformers for High-Resolution Image Synthesis2021
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Deformable ConvNets V2: More Deformable, Better Results2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal2018
- Direction-Aware Spatial Context Features for Shadow Detection2018
- DeshadowNet: A Multi-context Embedding Deep Network for Shadow Removal2017