HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation
Preprint
Abstract
Controllable human image generation (HIG) has numerous real-life applications. State-of-the-art solutions, such as ControlNet and T2I-Adapter, introduce an additional learnable branch on top of the frozen pre-trained stable diffusion (SD) model, which can enforce various conditions, including skeleton guidance of HIG. While such a plug-and-play approach is appealing, the inevitable and uncertain conflicts between the original images produced from the frozen SD branch and the given condition incur significant challenges for the learnable branch, which essentially conducts image feature editing for condition enforcement. In this work, we propose a native skeleton-guided diffusion model for controllable HIG called HumanSD. Instead of performing image editing with dual-branch diffusion, we fine-tune the original SD model using a novel heatmap-guided denoising loss. This strategy effectively and efficiently strengthens the given skeleton condition during model training while mitigating the catastrophic forgetting effects. HumanSD is fine-tuned on the assembly of three large-scale human-centric datasets with text-image-pose information, two of which are established in this work. As shown in Figure 1, HumanSD outperforms ControlNet in terms of accurate pose control and image quality, particularly when the given skeleton guidance is sophisticated.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer2023
- GLIGEN: Open-Set Grounded Text-to-Image Generation2023
- SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos2022
- Verbal-Person Nets: Pose-Guided Multi-Granularity Language-to-Person GenerationIEEE Transactions on Neural Networks and Learning Systems, 2022
- Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation2021
- Towards Fine-Grained Human Pose Transfer With Detail Replenishing NetworkIEEE Transactions on Image Processing, 2021