CMRPPGFormer: 3-D Spatio-Temporal Convolutional Modulation Transformer Network for Remote Heart Rate Estimation
- 6 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 24 (19) , 30275-30286
Abstract
Most of the existing remote heart rate (HR) estimation methods based on deep learning networks use convolutional neural networks which excel at local feature extraction. This paper proposes a HR estimation method based on a vision transformer network named as CMRPPGFormer. The basic idea of the proposed method is to extract precise remote photoplethysmography (rPPG) signals from facial videos by the CMRPPGFormer and then calculate relevant indicators of HR by specific post-processing on rPPG signals. The CMRPPGFormer consists of the convolutional stem block, the feature fusion block and the rPPG predictor block. The feature fusion block, the key of the CMRPPGFormer, can effectively extract and fuse local and global features of rPPG signals, in which a three dimensional (3D) spatio-temporal convolutional modulation transformer (3D-ST-CMTransformer) model is proposed. The 3D-ST-CMTransformer extracts global features of rPPG signals by its 3D convolution modulation attention layer and feed-forward network effectively. Experiment results on two public datasets, PURE and UBFC-rPPG, validate the better performance of the proposed method compared to the state-of-the-art baseline methods. Particularly, CMRPPGFormer obtains 0.39 bpm MAE, 0.64 bpm RMSE, 0.99 r value of Pearson’s correlation coefficient and 0.59 bpm MAE, 1.77 bpm RMSE, 0.99 r value of Pearson’s correlation coefficient on the PURE and UBFC-rPPG datasets, respectively. Compared to the baseline methods, it achieves the lowest MAE and RMSE, and the highest r.Keywords
Funding Information
- National Natural Science Foundation of China (61772088)
This publication has 42 references indexed in Scilit:
- TransPPG: two-stream transformer for remote heart rate estimateCCF Transactions on Pervasive Computing and Interaction, 2024
- PhysFormer++: Facial Video-Based Physiological Measurement with SlowFast Temporal Difference TransformerInternational Journal of Computer Vision, 2023
- DCGNN: a single-stage 3D object detection network based on density clustering and graph neural networkComplex & Intelligent Systems, 2022
- Instantaneous Physiological Estimation Using Video Transformers2022
- Contrast-Phys: Unsupervised Video-Based Remote Physiological Measurement via Spatiotemporal Contrast2022
- Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-learner2020
- New insights on super-high resolution for video-based heart rate estimation with a semi-blind source separation methodComputers in Biology and Medicine, 2020
- Unsupervised skin tissue segmentation for remote photoplethysmographyPattern Recognition Letters, 2019
- DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks2018
- Remote Heart Rate Measurement from Face Videos under Realistic Situations2014