HanDiffuser: Text-to-Image Generation with Realistic Hand Appearances
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2468-2479
Abstract
Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. To generate images with realistic hands, we propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process. HanDiffuser consists of two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and MANO-Hand parameters from input text prompts, and a Text-Guided Hand-Params-to-Image diffusion model to synthesize images by conditioning on the prompts and hand parameters generated by the previous component. We incorporate multiple aspects of hand representation, including 3D shapes and joint-level finger positions, orientations and articulations, for robust learning and reliable performance during inference. We conduct extensive quantitative and qualitative experiments and perform user studies to demonstrate the efficacy of our method in generating images with high-quality hands. Project page: https://supreethn.github.io/research/handiffuser/index.htmlKeywords
All Related Versions
Funding Information
- US National Science Foundation (DUE-2055406)
This publication has 44 references indexed in Scilit:
- CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes2022
- TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement2022
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors2022
- TEMOS: Generating Diverse Human Motions from Textual Descriptions2022
- BABEL: Bodies, Action and Behavior with English Labels2021
- Language2Pose: Natural Language Grounded Pose Forecasting2019
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis2019
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- The KIT Motion-Language DatasetBig Data, 2016
- Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signersComputer Vision and Image Understanding, 2015