Changchen Zhao

Remote photoplethysmography (rPPG) has long been an active research topic. Existing rPPG approaches achieve high accuracy of heart rate extraction, as long as the user is relatively close to the camera (typically, less than 1 meter distance). This article investigates the performance of existing rPPG approaches under the long-distance recording conditions and proposes a novel Projection of Rotated Orthogonal Bases in POS (ProPOS) algorithm for heart rate extraction. A set of orthogonal projection bases is generated around the original plain of POS algorithm. The raw measurement traces are projected on these bases and the final output signal is obtained by a designed SNR selection criterion. The long-distance rPPG (LD-rPPG) dataset is established for long-distance rPPG research by varying the recording distance from 3 m-30 m. Extensive experiments are performed in comparison with existing approaches. Experiments show that videos recorded by HikVision DS-V108 and Logitech C920 cameras contain a certain amount of physiological signal whereas the videos recorded by HikVision DS-U102D and Mercury cameras contain little physiological signal. Using zoom lenses is beneficial to improve the rPPG measurement accuracy under long-distance conditions.

Camera-based vital signs monitoring in recent years has attracted more and more researchers and the results are promising. However, a few research works focus on heart rate extraction under extremely low illumination environments. In this paper, we propose a novel framework for remote heart rate estimation under low-light conditions. This method uses singular spectrum analysis (SSA) to decompose the filtered signal into several reconstructed components. A spectral masking algorithm is utilized to refine the preliminary candidate components on the basis of a reference heart rate. The contributive components are fused into the final pulse signal. To evaluate the performance of our framework in low-light conditions, the proposed approach is tested on a large-scale multi-illumination HR dataset (named MIHR). The test results verify that the proposed method has stronger robustness to low illumination than state-of-the-art methods, effectively improving the signal-to-noise ratio and heart rate estimation precision. We further perform experiments on the PUlse RatE detection (PURE) dataset which is recorded under normal light conditions to demonstrate the generalization of our method. The experiment results show that our method can stably detect pulse rate and achieve comparative results. The proposed method pioneers a new solution to the remote heart rate estimation in low-light conditions.

Vision-based physiological monitoring is an emerging technology that enables a more convenient access of cardiovascular health status in many medical industrial applications. This article aims to achieve efficient and accurate identification of pulse waveforms by proposing a weighted temporally consistent 3-D (WTC3D) convolution, in which a spatial weight template is incorporated between the spatial and temporal kernels as a constraint for the temporal kernel. WTC3D employs a temporal kernel to keep temporal consistency and a spatial weight template to impose spatial diversity during the remote photoplethysmography (rPPG) feature learning. A WTC3D-based network with a hybrid loss function is designed for pulse prediction. Experiments on three datasets demonstrate the effectiveness of the proposed approach. By considering the temporal propagation characteristics of the pulse signal in the video, WTC3D convolution not only enables efficient pulse feature learning, but also advances the deployment of rPPG networks on source-limited Internet of medical things devices.

Although there are many wearable sensors that make the acquisition of multi-modality data easier, effective feature extraction and fusion of the data is still challenging for lower limb locomotion mode recognition. In this article, a novel neural network is proposed for accurate prediction of five common lower limb locomotion modes including level walking, ramp ascent, ramp descent, stair ascent, and stair descent. First, the encoder-decoder structure is employed to enrich the channel diversity for the separation of the useful patterns from combined patterns. Second, a self-attention based cross-modality interaction module is proposed, which enables bilateral information flow between two encoding paths to fully exploit the interdependencies and to find complementary information between modalities. Third, a multi-modality fusion module is designed where the complementary features are fused by a channel-wise weighted summation whose coefficients are learned end-to-end. A benchmark dataset is collected from 10 health subjects containing EMG and IMU signals and five locomotion modes. Extensive experiments are conducted on one publicly available dataset ENABL3S and one self-collected dataset. The results show that the proposed method outperforms the compared methods with higher classification accuracy. The proposed method achieves a classification accuracy of 98.25 $\%$ on ENABL3S dataset and 95.51 $\%$ on the self-collected dataset. Note to Practitioners —This article aims to solve the real challenges encountered when intelligent recognition algorithms are applied in wearable robots: how to effectively and efficiently fuse the multi-modality data for better decision-making. First, most existing methods directly concatenate the multi-modality data, which increases the data dimensionality and brings computational burden. Second, existing recognition neural networks continuously compress the feature size such that the discriminative patterns are submerged in the noise and thus difficult to be identified. This research decomposes the mixed input signals on the channel dimension such that the useful patterns can be separated. Moreover, this research employs self-attention mechanism to associate correlations between two modalities and use this correlation as a new feature for subsequent representation learning, generating new, compact, and complementary features for classification. We demonstrate that the proposed network achieves 98.25 $\%$ accuracy and 3.5 ms prediction time. We anticipate that the proposed network could be a general scientific and practical methodology of multi-modality signal fusion and feature learning for intelligent systems.

The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO₂) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.

With the improvement of sensor technology and significant algorithmic advances, the accuracy of remote heart rate monitoring technology has been significantly improved. Despite of the significant algorithmic advances, the performance of rPPG algorithm can degrade in the long-term, high-intensity continuous work occurred in evenings or insufficient light environments. One of the main challenges is that the lost facial details and low contrast cause the failure of detection and tracking. Also, insufficient lighting in video capturing hurts the quality of physiological signal. In this paper, we collect a large-scale dataset that was designed for remote heart rate estimation recorded with various illumination variations to evaluate the performance of the rPPG algorithm (Green, ICA, and POS). We also propose a low-light enhancement solution (technical solution) for remote heart rate estimation under the low-light condition. Using collected dataset, we found 1) face detection algorithm cannot detect faces in video captured in low light conditions; 2) A decrease in the amplitude of the pulsatile signal will lead to the noise signal to be in the dominant position; and 3) the chrominance-based method suffers from the limitation in the assumption about skin-tone will not hold, and Green and ICA method receive less influence than POS in dark illuminance environment. The proposed solution for rPPG process is effective to detect and improve the signal-to-noise ratio and precision of the pulsatile signal.

Recent studies have demonstrated the benefit of extracting and fusing pulse signals from multi-scale region-of-interests (ROIs). However, these methods suffer from heavy computational load. This paper aims to effectively utilize multi-scale rPPG features with a more compact architecture. Inspired by recent research works exploring two-path architecture that leverages global and local information with bidirectional bridge in between. This paper designs a novel architecture Global-Local Interaction and Supervision Network (GLISNet), which uses a local path to learn representations in the original scale and a global path to learn representations in the other scale capturing multi-scale information. A light-weight rPPG signal generation block is attached to the output of each path that maps the pulse representation to the pulse output. A hybrid loss function is utilized enabling the local and global representations to learn directly from the training data. Extensive experiments are conducted on two publicly available datasets, and results demonstrate that GLISNet achieves superior performance in terms of signal-to-noise ratio (SNR), mean absolute error (MAE), and root mean squared error (RMSE). In terms of SNR, GLISNet has an increase of 4.41% compared with the second best algorithm PhysNet on PURE dataset. The MAE has a decrease of 13.16% compared with the second best algorithm DeeprPPG on UBFC-rPPG dataset. The RMSE has a decrease of 26.29% compared with the second best algorithm PhysNet on UBFC-rPPG dataset. Experiments on MIHR dataset demonstrates the robustness of GLISNet under low-light environment.

Remote photoplethysmography (rPPG) has been an active research topic in recent years. While most existing methods are focusing on eliminating motion artifacts in the raw traces obtained from single-scale region-of-interest (ROI), it is worth noting that there are some noise signals that cannot be effectively separated in single-scale space but can be separated more easily in multi-scale space. In this paper, we analyze the distribution of pulse signal and motion artifacts in different layers of a Gaussian pyramid. We propose a method that combines multi-scale analysis and neural network for pulse extraction in different scales, and a layer-wise attention mechanism to adaptively fuse the features according to signal strength. In addition, we propose spatial-temporal joint attention module and channel-temporal joint attention module to learn and exaggerate pulse features in the joint spaces, respectively. The proposed remote pulse extraction network is called Joint Attention and Multi-Scale fusion Network (JAMSNet). Extensive experiments have been conducted on two publicly available datasets and one self-collected dataset. The results show that the proposed JAMSNet shows better performance than state-of-the-art methods.