rPPG Estimation: Vision Transformer With 3-D Temporal Central Difference

Authors:
Publisher website
Google Scholar
Abstract
Remote photoplethysmography (rPPG) has gained increasing importance, especially during and after the COVID-19 pandemic, for its ability to estimate heart rate by analyzing subtle variations in skin color without physical contact. This non-invasive and practical method relies on capturing changes in pixel intensity through RGB video or near-infrared imaging. In this study, we propose a novel hybrid model that leverages the feed-forward integration of 3D Convolutional Neural Networks (3D-CNNs) and Video Vision Transformers (ViViTs) to enhance rPPG estimation. The 3D-CNNs first capture local spatiotemporal features, using Temporal Central Difference Convolution (3DCDC-T) and Convolutional Block Attention Module (CBAM). These local features are then passed to the Video Vision Transformer, where Multi-Head Self-Attention (MHSA) captures global contextual relationships and long-range dependencies across frames, enabling more effective representation of complex temporal dynamics. This sequential learning allows the model to progressively refine features from local to global, ensuring more consistent and coherent feature extraction. Our feed-forward approach also improves computational efficiency by reducing the dimensionality of the input data before global attention processing, making it particularly effective in datalimited environments. Through comprehensive experiments, we show that our hybrid approach outperforms state-of-the-art methods across multiple public datasets, achieving a 22.55% improvement in MAE and a 55.80% improvement in RMSE on the UBFC-rPPG dataset, demonstrating superior feature progression and generalization in rPPG and heart rate estimation tasks.
Funding Information
  • Natural Sciences and Engineering Research Council (NSERC) of Canada
  • Fonds de recherche du Québec-Santé

This publication has 32 references indexed in Scilit: