Reconstructing detailed dynamic face geometry from monocular video
Open Access
- 1 November 2013
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 32 (6) , 1-10
Abstract
Detailed facial performance geometry can be reconstructed using dense camera and light setups in controlled studios. However, a wide range of important applications cannot employ these approaches, including all movie productions shot from a single principal camera. For post-production, these require dynamic monocular face capture for appearance modification. We present a new method for capturing face geometry from monocular video. Our approach captures detailed, dynamic, spatio-temporally coherent 3D face geometry without the need for markers. It works under uncontrolled lighting, and it successfully reconstructs expressive motion including high-frequency face detail such as folds and laugh lines. After simple manual initialization, the capturing process is fully automatic, which makes it versatile, lightweight and easy-to-deploy. Our approach tracks accurate sparse 2D features between automatically selected key frames to animate a parametric blend shape model, which is further refined in pose, expression and shape by temporally coherent optical flow and photometric stereo. We demonstrate performance capture results for long and complex face sequences captured indoors and outdoors, and we exemplify the relevance of our approach as an enabling technology for model-based face editing in movies and video, such as adding new facial textures, as well as a step towards enabling everyone to do facial performance capture with a single affordable camera.Keywords
This publication has 30 references indexed in Scilit:
- Realtime performance-based facial animationACM Transactions on Graphics, 2011
- High-quality passive facial performance capture using anchor framesACM Transactions on Graphics, 2011
- Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisitionACM Transactions on Graphics, 2011
- High resolution passive facial performance captureACM Transactions on Graphics, 2010
- Example-based facial riggingACM Transactions on Graphics, 2010
- Multi-scale capture of facial geometry and motionACM Transactions on Graphics, 2007
- Efficiently combining positions and normals for precise 3D geometryACM Transactions on Graphics, 2005
- Face transfer with multilinear modelsACM Transactions on Graphics, 2005
- High Resolution Acquisition, Learning and Transfer of Dynamic 3‐D Facial ExpressionsComputer Graphics Forum, 2004
- Spacetime facesACM Transactions on Graphics, 2004