Self-Supervised RGB-NIR Fusion Video Vision Transformer Framework for rPPG Estimation
Open Access
- 28 October 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 71 (00189456) , 1-10
Abstract
Remote photoplethysmography (rPPG) is a technology that can estimate non-contact heart rate (HR) using facial videos. Estimating rPPG signals requires low cost, and thus, it is widely used for non-contact health monitoring. Recent HR estimation studies based on rPPG heavily rely on the supervised feature learning on normal RGB videos. However, the RGB-only methods are significantly affected by head movements and various illumination conditions, and it is difficult to obtain large-scale labeled data for rPPG in order to determine the performance of supervised learning methods. To address these problems, we present the first of its kind self-supervised transformer-based fusion learning framework for rPPG estimation. In our study, we propose an end-to-end Fusion Video Vision Transformer (Fusion ViViT) network that can extract long-range local and global spatiotemporal features from videos and convert them into video sequences to enhance the rPPG representation. In addition, the self-attention of the transformer integrates the spatiotemporal representations of complementary RGB and near-infrared (NIR), which, in turn, enable robust HR estimation even under complex conditions. We use contrastive learning as a self-supervised learning scheme. We evaluate our framework on public datasets containing both RGB, NIR videos and physiological signals. The result of near-instant HR (approximately 6 s) estimation on the large-scale rPPG dataset with various scenarios, was 14.86 of RMSE, which was competitive with the state-of-the-art accuracy of average HR (approximately 30 s). Furthermore, transfer learning results on the driving rPPG dataset showed a stable HR estimation performance with 16.94 of RMSE, demonstrating that our framework can be utilized in the real world.Keywords
Funding Information
- National Research Foundation of Korea
- Korea Government [Ministry of Science and ICT (MSIT)] (2021R1F1A1052389)
- MSIT KOREA
- ICT Challenge and Advanced Network (ICAN of HRD) Program (IITP-2022-RS-2022-00156299)
- Institute of Information Communications Technology Planning Evaluation
This publication has 36 references indexed in Scilit:
- Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situationPLOS ONE, 2018
- Residual Attention Network for Image Classification2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Photoplethysmography Revisited: From Contact to Noncontact, From Point to ImagingIEEE Transactions on Biomedical Engineering, 2015
- Motion Robust Remote-PPG in InfraredIEEE Transactions on Biomedical Engineering, 2015
- Non-contact video-based pulse rate measurement on a mobile service robot2014
- Physiological parameter monitoring of drivers based on video data and independent vector analysis2014
- Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit — A pilot studyEarly Human Development, 2013
- A Multimodal Database for Affect Recognition and Implicit TaggingIEEE Transactions on Affective Computing, 2011
- Rapid object detection using a boosted cascade of simple features2005