Towards Real-Time Facial Landmark Detection in Depth Data Using Auxiliary Information
Open Access
Abstract
Modern facial motion capture systems employ a two-pronged approach for capturing and rendering facial motion. Visual data (2D) is used for tracking the facial features and predicting facial expression, whereas Depth (3D) data is used to build a series of expressions on 3D face models. An issue with modern research approaches is the use of a single data stream that provides little indication of the 3D facial structure. We compare and analyse the performance of Convolutional Neural Networks (CNN) using visual, Depth and merged data to identify facial features in real-time using a Depth sensor. First, we review the facial landmarking algorithms and its datasets for Depth data. We address the limitation of the current datasets by introducing the Kinect One Expression Dataset (KOED). Then, we propose the use of CNNs for the single data stream and merged data streams for facial landmark detection. We contribute to existing work by performing a full evaluation on which streams are the most effective for the field of facial landmarking. Furthermore, we improve upon the existing work by extending neural networks to predict into 3D landmarks in real-time with additional observations on the impact of using 2D landmarks as auxiliary information. We evaluate the performance by using Mean Square Error (MSE) and Mean Average Error (MAE). We observe that the single data stream predicts accurate facial landmarks on Depth data when auxiliary information is used to train the network. The codes and dataset used in this paper will be made available.Keywords
This publication has 27 references indexed in Scilit:
- Learning Deep Representation for Face Alignment with Auxiliary AttributesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification2014
- KinectFaceDB: A Kinect Database for Face RecognitionIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2014
- FaceWarehouse: A 3D Facial Expression Database for Visual ComputingIEEE Transactions on Visualization and Computer Graphics, 2013
- Deep Convolutional Network Cascade for Facial Point Detection2013
- An RGB-D Database Using Microsoft's Kinect for Windows for Face Detection2012
- Random Forests for Real Time 3D Face AnalysisInternational Journal of Computer Vision, 2012
- Overview of the Face Recognition Grand Challenge2005
- Long Short-Term MemoryNeural Computation, 1997
- An evolutionary algorithm that constructs recurrent neural networksIEEE Transactions on Neural Networks, 1994