A Lightweight 3DMM-CNN Pipeline for Real-Time Single-Image 3D Face Reconstruction: Prototyping Personalised Avatars for Extended Reality Applications

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Personalised three-dimensional (3D) facial avatars underpin a wide range of immersive virtual, augmented, and mixed reality (XR) experiences, yet conventional 3D capture pipelines remain prohibitively expensive and computationally demanding for prototype-stage XR applications. This study presents and evaluates a lightweight hybrid 3D Morphable Model–Convolutional Neural Network (3DMM-CNN) pipeline that reconstructs an animation-ready 3D facial mesh from a single unconstrained RGB photograph and exposes it through an interactive prototype with native export to XR-ready asset formats. A four-channel ResNet-50 backbone fuses RGB pixels with a landmark-mask channel, regresses the 3DMM shape, expression, pose, and illumination parameters, and is refined through a multi-task loss that combines 3D parameter regression, 2D landmark consistency, and image-to-mesh-to-image cycle consistency. The model is trained on a curated 2000-image subset of the LFW-People corpus and evaluated under four yaw-angle strata. The results indicate that on a held-out 400-image test set, the pipeline attains R2 = 0.854, MSE = 0.022, Pearson r = 0.92, and MAPE = 10.6%, with a single-frame inference latency of 35 ms on a commodity RTX-class GPU. Robustness to head rotation improves by 29.9% at extreme poses (60–90° yaw) compared with a single-modality baseline. A Blender-integrated prototype successfully exports the reconstructed mesh as a deformation-ready asset for Unity- and Unreal-based XR engines. The proposed pipeline offers a cost-effective, real-time-capable component for XR avatar prototyping, lowering the entry barrier for small studios, immersive-learning developers, and AR/MR telepresence research. On the standard AFLW2000-3D benchmark, the pipeline additionally attains a Normalised Mean Error of 2.47% and a full-vertex reconstruction error of 1.50%, which is competitive with published lightweight baselines while retaining sub-50 ms inference latency.

This publication has 35 references indexed in Scilit: