HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpainting
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 7085-7093
Abstract
Diffusion models have achieved remarkable success in generating realistic images but suffer from generating accurate human hands, such as incorrect finger counts or irregular shapes. This difficulty arises from the complex task of learning the physical structure and pose of hands from training images, which involves extensive deformations and occlusions. For correct hand generation, our paper introduces a lightweight post-processing solution called HandRefiner. HandRefiner employs a conditional inpainting approach to rectify malformed hands while leaving other parts of the image untouched. We leverage the hand mesh reconstruction model that consistently adheres to the correct number of fingers and hand shape, while also being capable of fitting the desired hand pose in the generated image. Given a generated failed image due to malformed hands, we utilize ControlNet modules to re-inject such correct hand information. Additionally, we uncover a phase transition phenomenon within ControlNet as we vary the control strength. It enables us to take advantage of more readily available synthetic data without suffering from the domain gap between realistic and synthetic hands. Experiments demonstrate that HandRefiner can significantly improve the generation quality quantitatively and qualitatively. The code is available at https://github.com/wenquanlu/HandRefiner.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- HanDiffuser: Text-to-Image Generation with Realistic Hand Appearances2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Affordance Diffusion: Synthesizing Hand-Object Interactions2023
- RePaint: Inpainting using Denoising Diffusion Probabilistic Models2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Fake it till you make it: face analysis in the wild using synthetic data alone2021
- Learning to Recover 3D Scene Shape from a Single Image2021
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images2019
- MirrorGAN: Learning Text-To-Image Generation by Redescription2019
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017