Abstract
Most of the existing remote heart rate (HR) estimation methods based on deep learning networks use convolutional neural networks which excel at local feature extraction. This paper proposes a HR estimation method based on a vision transformer network named as CMRPPGFormer. The basic idea of the proposed method is to extract precise remote photoplethysmography (rPPG) signals from facial videos by the CMRPPGFormer and then calculate relevant indicators of HR by specific post-processing on rPPG signals. The CMRPPGFormer consists of the convolutional stem block, the feature fusion block and the rPPG predictor block. The feature fusion block, the key of the CMRPPGFormer, can effectively extract and fuse local and global features of rPPG signals, in which a three dimensional (3D) spatio-temporal convolutional modulation transformer (3D-ST-CMTransformer) model is proposed. The 3D-ST-CMTransformer extracts global features of rPPG signals by its 3D convolution modulation attention layer and feed-forward network effectively. Experiment results on two public datasets, PURE and UBFC-rPPG, validate the better performance of the proposed method compared to the state-of-the-art baseline methods. Particularly, CMRPPGFormer obtains 0.39 bpm MAE, 0.64 bpm RMSE, 0.99 r value of Pearson’s correlation coefficient and 0.59 bpm MAE, 1.77 bpm RMSE, 0.99 r value of Pearson’s correlation coefficient on the PURE and UBFC-rPPG datasets, respectively. Compared to the baseline methods, it achieves the lowest MAE and RMSE, and the highest r.
Funding Information
  • National Natural Science Foundation of China (61772088)