When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 6979-6988
Abstract
ControlNet excels at creating content that closely matches precise contours in user-provided masks. However, when these masks contain noise, as a frequent occurrence with non-expert users, the output would include unwanted artifacts. This paper first highlights the crucial role of controlling the impact of these inexplicit masks with diverse deterioration levels through in-depth analysis. Subsequently, to enhance controllability with inexplicit masks, an advanced Shape-aware ControlNet consisting of a deterioration estimator and a shape-prior modulation block is devised. The deterioration estimator assesses the deterioration factor of the provided masks. Then this factor is used in a modulation block to adaptively adjust the model's contour-following ability, which helps it dismiss the noise part in the inexplicit masks. Extensive experiments prove its effectiveness in encouraging ControlNet to interpret inaccurate spatial conditions robustly rather than blindly following the given contours, suitable for diverse kinds of conditions. We showcase application scenarios like modifying shape priors and composable shape-controllable generation. Codes are available at github.Keywords
All Related Versions
Funding Information
- The National Natural Science Foundation of China (U23B2048, 62076186 and 62225113)
- the Innovative Research Group Project of Hubei Province (2024AFA017)
- the national key Research and Development Program of China (2023YFC2705700)
This publication has 18 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- SpaText: Spatio-Textual Representation for Controllable Image Generation2023
- GLIGEN: Open-Set Grounded Text-to-Image Generation2023
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset TransferIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- LVIS: A Dataset for Large Vocabulary Instance Segmentation2019
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Holistically-Nested Edge Detection2015
- Microsoft COCO: Common Objects in Context2014