TECA: Text-Guided Generation and Editing of Compositional 3D Avatars
Authors:
- 18 March 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1520-1530
Abstract
Our goal is to create a realistic 3D facial avatar with hair and accessories using only a text description. While this challenge has attracted significant recent interest, existing methods either lack realism, produce unrealistic shapes, or do not support editing, such as modifications to the hairstyle. We argue that existing methods are limited because they employ a monolithic modeling approach, using a single representation for the head, face, hair, and accessories. Our observation is that the hair and face, for example, have very different structural qualities that benefit from different representations. Building on this insight, we generate avatars with a compositional model, in which the head, face, and upper body are represented with traditional 3D meshes, and the hair, clothing, and accessories with neural radiance fields (NeRF). The model-based mesh representation provides a strong geometric prior for the face region, improving realism while enabling editing of the person’s appearance. By using NeRFs to represent the remaining components, our method is able to model and synthesize parts with complex geometry and appearance, such as curly hair and fluffy scarves. Our novel system synthesizes these high-quality compositional avatars from text descriptions. Specifically, we generate a face image using text, fit a parametric shape model to it, and inpaint texture using diffusion models. Conditioned on the generated face, we sequentially generate style components such as hair or clothing using Score Distillation Sampling (SDS) with guidance from CLIPSeg segmentations. However, this alone is not sufficient to produce avatars with a high degree of realism. Consequently, we introduce a hierarchical approach to refine the non-face regions using a BLIP-based loss combined with SDS. The experimental results demonstrate that our method, Text-guided generation and Editing of Compositional Avatars (TECA), produces avatars that are more realistic than those of recent methods while being editable because of their compositional nature. For example, our TECA enables the seamless transfer of compositional features like hairstyles, scarves, and other accessories between avatars. This capability supports applications such as virtual try-on. The code and generated avatars will be publicly available for research purposes at yfeng95.github.io/teca.Keywords
This publication has 54 references indexed in Scilit:
- Towards Metrical Reconstruction of Human Faces2022
- Neural Strands: Learning Hair Geometry and Appearance from Multi-view Images2022
- Structural Causal 3D Reconstruction2022
- Zero-Shot Text-Guided Object Generation with Dream Fields2022
- CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields2022
- NeuralHDHair: Automatic High-fidelity Hair Modeling from a Single Image Using Implicit Neural Representations2022
- Collaborative Regression of Expressive Bodies using Moderation2021
- Expressive Body Capture: 3D Hands, Face, and Body From a Single Image2019
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network2018