TeCH: Text-Guided Reconstruction of Lifelike Clothed Humans
Authors:
Top Cited Papers
- 18 March 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1531-1542
Abstract
Despite recent research advancements in reconstructing clothed humans from a single image, accurately restoring the “unseen regions” with high-level details remains an unsolved challenge that lacks attention. Existing methods often generate overly smooth back-side surfaces with a blurry texture. But how to effectively capture all visual attributes of an individual from a single image, which are sufficient to reconstruct unseen areas (e.g. the back view)? Motivated by the power of foundation models, TeCH reconstructs the 3D human by leveraging 1) descriptive text prompts (e.g. garments, colors, hairstyles) which are automatically generated via a garment parsing model and Visual Question Answering (VQA), 2) a personalized fine-tuned Text-to-Image diffusion model (T2I) which learns the “indescribable” appearance. To represent high-resolution 3D clothed humans at an affordable cost, we propose a hybrid 3D representation based on DMTet, which consists of an explicit body shape grid and an implicit distance field. Guided by the descriptive prompts + personalized T2I diffusion model, the geometry and texture of the 3D humans are optimized through multi-view Score Distillation Sampling (SDS) and reconstruction losses based on the original observation. TeCH produces high-fidelity 3D clothed humans with consistent & delicate texture, and detailed full-body geometry. Quantitative and qualitative experiments demonstrate that TeCH outperforms the state-of-the-art methods in terms of reconstruction accuracy and rendering quality. The code will be publicly available for research purposes at huangyangyi.github.io/TeCHKeywords
All Related Versions
Funding Information
- Ministry of Education
- National Natural Science Foundation of China
This publication has 109 references indexed in Scilit:
- AvatarGen: A 3D Generative Model for Animatable Human Avatars2023
- HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling2022
- Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations2022
- BCNet: Learning Body and Cloth Shape from a Single Image2020
- Monocular Expressive Body Regression Through Body-Driven Attention2020
- MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video2020
- Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild2020
- 360-Degree Textures of People in Clothing from a Single Image2019
- Detailed Human Avatars from Monocular Video2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017