Matthias Nießner

This state‐of‐the‐art report provides an overview of controllable 3D human avatar creation. We describe current 3D avatar systems, which typically consist of three stages: (i) learning priors of human appearance and motion, (ii) creating a personalized avatar, and (iii) animating the avatar. To limit the scope, we focus on the prior learning and avatar creation stages. We define current avatar representations and introduce a taxonomy that categorizes existing work along multiple axes, including body regions and employed priors. We review methods for full‐body and head avatars, as well as layered representations that decompose the body into components such as hands, hair, and garments. Finally, we outline common underlying principles, reference key literature for newcomers, and discuss open challenges and future research directions.

We introduce Motion2VecSets, a 4D diffusion model for dynamic surface mesh generation from various ambiguous observations, including a sequence of RGB images, sparse and partial point clouds, and low-resolution voxel grids. While recent methods using neural field representations have shown success in modeling non-rigid objects, conventional feed-forward architectures struggle with noisy, partial, or sparse observations due to their deterministic nature. To address the inherent one-to-many mapping problem, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based priors provide more plausible and diverse reconstructions under ambiguous conditions. Instead of relying on global latent codes, we represent 4D dynamics using latent sets. This novel 4D representation captures local shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalization capacity to unseen motions and identities. For temporally coherent tracking, we jointly denoise latent sets across frames and enable cross-frame information exchange. To reduce computational cost, we design an interleaved spatial-temporal attention block that alternately aggregates deformation latents along spatial and temporal dimensions. Extensive experiments on datasets of humans, animals, and articulated objects demonstrate that Motion2VecSets outperforms prior methods in reconstructing and tracking non-rigid deformations from various imperfect observations. Our implementation is available at https://vveicao.github.io/projects/Motion2VecSets/.

The refurbishment and energy-efficient renovation of existing buildings, specifically those before 1945, with complex architectural building elements, require a level of building information that is often unavailable, incomplete or imprecise. As they represent a large percentage of current building stock (up to 25% of all buildings), it is crucial to address such an issue, as such buildings are ideal targets for renovation and energy retrofitting projects. This paper presents a conceptual pipeline, developed through the NERF2BIM research project, focusing on an Artificial Intelligence (AI)-supported holistic pipeline for the creation of as-is Building Information Modeling (BIM) models of existing buildings, expanding upon existing methodologies with the embedding of knowledge-driven semi-automatic detailing-on-demand task. The proposed pipeline integrates uncertainty-aware spatial capture, semantic interpretation and reconstruction, and knowledge-based reasoning within a BIM-oriented workflow. The paper provides an overview of current advancements in the respective aspects of the pipeline, highlighting current gaps. The proposed conceptual pipeline aims at addressing these issues through novel applications of AI and knowledge-driven solutions. Key contributions include: (1) a conceptual approach in addressing the imprecision of more sustainable data gathering approaches; (2) a context-aware BIM reconstruction process, providing multiple data output types; and (3) a formalization of architectural and construction knowledge and its utilization in a detailing-on-demand approach of the reconstructed BIM models. Through the integration of uncertainty-aware data gathering, context-aware reconstructions and domain expertise into existing reconstruction pipelines, the proposed pipeline bridges the data gap for existing buildings, enabling more efficient and knowledge-driven renovation processes.

Human hair reconstruction is a challenging problem in computer vision, with growing importance for applications in virtual reality and digital human modeling. Recent advances in 3D Gaussians Splatting (3DGS) provide efficient and explicit scene representations that naturally align with the structure of hair strands. In this work, we extend the 3DGS framework to enable strand-level hair geometry reconstruction from multi-view images. Our multi-stage pipeline first reconstructs detailed hair geometry using a differentiable Gaussian rasterizer, then merges individual Gaussian segments into coherent strands through a novel merging scheme, and finally refines and grows the strands under photometric supervision. While existing methods typically evaluate reconstruction quality at the geometric level, they often neglect the connectivity and topology of hair strands. To address this, we propose a new evaluation metric that serves as a proxy for assessing topological accuracy in strand reconstruction. Extensive experiments on both synthetic and real-world datasets demonstrate that our method robustly handles a wide range of hairstyles and achieves efficient reconstruction, typically completing within one hour. The project page can be found at: https://yimin-pan.github.io/hair-gs/