JHPFA-Net: Joint Head Pose and Facial Action Network for Driver Yawning Detection Across Arbitrary Poses in Videos
- 22 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 24 (11) , 11850-11863
Abstract
Yawning detection is a key means in driver fatigue detection, which suffers difficulties including head poses, facial expressions, illumination variations, occlusions, etc. Yet, most previous methods mainly focus on frontal faces, and are deficient to deal with different facial actions under arbitrary poses in the actual driving environment. In this study, we propose a novel Joint Head Pose and Facial Action Network (JHPFA-Net) for driver yawning detection across arbitrary poses in videos, with three main parts including a Geometric-based Key-frame Selection Module (GK-Module), a Face Frontalization with Warp Attention Module (FF-Module) and a dual-channel classifier for Head Pose $\&$ Facial Action Fusion Module (HF-Module). Firstly, the GK-Module is proposed to extract geometric vectors and to construct a two-stage judgment mechanism, with the purpose of dealing with frame redundancy and improving the efficiency of JHPFA-Net structure. Secondly, distinguished with existing methods, the FF-Module is proposed to synthesize photo-realistic frontal faces, which can be used for capturing the facial actions under arbitrary poses. Finally, the HF-Module is proposed to fuse head pose attributes and facial modalities together, for the purpose of achieving pose-invariant detection and improving accuracy. Extensive experiments show that the proposed JHPFA-Net achieves state-of-the-art results comparing with some representative methods on the public YawDD benchmark, and it performs well in real-time application.Keywords
Funding Information
- National Natural Science Foundation of China (U22A2058, 62176138, 62176136)
- shandong provincial key research and development program (2019JZZY010130, 2020CXGC010207)
This publication has 49 references indexed in Scilit:
- Facial video‐based detection of physical fatigue for maximal muscle activityIET Computer Vision, 2016
- Yawning Detection Using Embedded Smart CamerasIEEE Transactions on Instrumentation and Measurement, 2016
- Robust Statistical Face Frontalization2015
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Driver fatigue recognition based on facial expression analysis using local binary patternsOptik, 2015
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Yawn analysis with mouth occlusion detectionBiomedical Signal Processing and Control, 2015
- Adaptive Keyframe Selection for Video Summarization2015
- YawDD2014
- Boosted key-frame selection and correlated pyramidal motion-feature representation for human action recognitionPattern Recognition, 2013