Isotropic Self-Supervised Learning for Driver Drowsiness Detection With Attention-Based Multimodal Fusion
- 17 November 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 529-542
Abstract
Driver drowsiness is an important cause of traffic accidents. Many studies using computer vision techniques to detect driver drowsiness states, such as slow blinking, yawning, and nodding, have demonstrated excellent potential. Although existing studies have made significant progress, the number of samples in the training corpora is small, which makes it difficult for a model to learn effective drowsiness representations from images or videos. To address this issue, we develop an isotropic self-supervised learning (IsoSSL) approach to learn powerful representations of images without relying on human-provided annotations and propose an IsoSSL-MoCo model by combining IsoSSL with momentum contrast (MoCo). To exploit the complementarity of multimodal data, an attention-based multimodal fusion model is also proposed to fuse features from the eye, mouth, and optical flow of the head. Specifically, we first use the IsoSSL-MoCo model to pretrain the image encoders for the three modalities in other datasets. Then, these encoders are fine-tuned and integrated into the proposed fusion model. The feature vectors generated by the image encoders of the three modalities are fed into the recursive layer to extract temporal information. To capture the importance degrees of the effects of temporal features from the three modalities on drowsiness detection, an attention mechanism is introduced to automatically weigh the feature vectors from the recursive layer to improve detection accuracy. Finally, a vector representation is generated by the attention layer and is used to detect driver drowsiness states. Experimental results based on two challenging datasets show that our method outperforms the baseline methods and the latest existing methods.Keywords
Funding Information
- National Natural Science Foundation of China (61672068)
This publication has 60 references indexed in Scilit:
- A Review and Meta-Analysis of Multimodal Affect Detection SystemsACM Computing Surveys, 2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- Learning High-Level Feature by Deep Belief Networks for 3-D Model Retrieval and RecognitionIEEE Transactions on Multimedia, 2014
- YawDD2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Multimedia classification and event detection using double fusionMultimedia Tools and Applications, 2013
- A Duality Based Approach for Realtime TV-L 1 Optical Flow2007
- Early versus late fusion in semantic video analysis2005
- Histograms of Oriented Gradients for Human Detection2005
- Long Short-Term MemoryNeural Computation, 1997