Person Image Synthesis via Denoising Diffusion Model
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5968-5976
Abstract
The pose-guided person image generation task requires synthesizing photorealistic images of humans in arbitrary poses. The existing approaches use generative adversarial networks that do not necessarily maintain realistic textures or need dense correspondences that struggle to handle complex deformations and severe occlusions. In this work, we show how denoising diffusion models can be applied for high-fidelity person image synthesis with strong sample diversity and enhanced mode coverage of the learnt data distribution. Our proposed Person Image Diffusion Model (PIDM) disintegrates the complex transfer problem into a series of simpler forward-backward denoising steps. This helps in learning plausible source-to-target transformation trajectories that result in faithful textures and undistorted appearance details. We introduce a ‘texture diffusion module’ based on cross-attention to accurately model the correspondences between appearance and pose information available in source and target images. Further, we propose ‘disentangled classifier-free guidance’ to ensure close resemblance between the conditional inputs and the synthesized output in terms of both pose and appearance information. Our extensive results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios. We also show how our generated images can help in downstream tasks. Code is available at https://github.com/ankanbhunia/PIDM.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Cross-Domain Correspondence Learning for Exemplar-Based Image Translation2020
- Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis2019
- Dense Intrinsic Appearance Flow for Human Pose Transfer2019
- Progressive Pose Attention Transfer for Person Image Generation2019
- A Variational U-Net for Conditional Appearance and Shape Generation2018
- Deformable GANs for Pose-Based Human Image Generation2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017
- DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations2016
- Scalable Person Re-identification: A Benchmark2015