3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework Using Fully 3D Convolutional Neural Networks
Open Access
- 23 August 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 121655-121664
Abstract
The capability to perform facial analysis from video sequences has significant potential to positively impact in many areas of life. One such area relates to the medical domain to specifically aid in the diagnosis and rehabilitation of patients with facial palsy. With this application in mind, this paper presents an end-to-end framework, named 3DPalsyNet, for the tasks of mouth motion recognition and facial palsy grading. 3DPalsyNet utilizes a 3D CNN architecture with a ResNet backbone for the prediction of these dynamic tasks. Leveraging transfer learning from a 3D CNNs pre-trained on the Kinetics data set for general action recognition, the model is modified to apply joint supervised learning using center and softmax loss concepts. 3DPalsyNet is evaluated on a test set consisting of individuals with varying ranges of facial palsy and mouth motions and the results have shown an attractive level of classification accuracy in these tasks of 82% and 86% respectively. The frame duration and the loss function affect was studied in terms of the predictive qualities of the proposed 3DPalsyNet, where it was found shorter frame duration’s of 8 performed best for this specific task. Centre loss and softmax have shown improvements in spatio-temporal feature learning than softmax loss alone, this is in agreement with earlier work involving the spatial domain.Keywords
All Related Versions
Funding Information
- Engineering and Physical Sciences Research Council (EP/P009727/1)
This publication has 26 references indexed in Scilit:
- Video-based emotion recognition using CNN-RNN and C3D hybrid networks2016
- Paralysie faciale chez l’enfantAnnales de Chirurgie Plastique Esthétique, 2016
- A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression RecognitionIEEE Transactions on Affective Computing, 2015
- Automatic evaluation of the degree of facial nerve paralysisMultimedia Tools and Applications, 2015
- Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression RecognitionPLOS ONE, 2015
- Automatic recognition of facial movement for paralyzed faceBio-Medical Materials and Engineering, 2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Comprehensive Facial Rehabilitation Improves Function in People With Facial Paralysis: A 5-Year Experience at the Massachusetts Eye and Ear InfirmaryPhysical Therapy, 2010
- Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial ExpressionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2007
- Facial expression recognition from video sequences: temporal and static modelingComputer Vision and Image Understanding, 2003