Zhaolin Qiu

Heart rate (HR) is one of the most important physiological signals of a person. Remote HR estimation derives subtle color variations from one’s facial video captured by a camera. The periodic signal caused by heart activity is then extracted from the color variations, resulting in heart rate. The advantages of remote methods show in their contactlessness, comfortableness, and impalpability. Due to the very subtle color changes of the face, remote methods are quite sensitive to illumination variations and motion artifacts. In current studies, researchers have found that the results achieved from experiments conducted on different color spaces vary significantly, suggesting that the choice of color space has an important impact on HR estimation. However, it is difficult to propose an optimum color space that is appropriate for the HR estimation task. Additionally, there is a lack of specialized training strategies for color space projection tasks as well as combinations with the state-of-the-art mechanisms. In this article, we propose a HR estimating network with adaptive color space transformation, abbreviated as CoSTHR, to address these challenges. The network uses color space transformation (CoST) layers to learn the optimum color space for HR estimation task, based on which the attention convolutional neural network can focus on the informative channels and extract the HR. Specialized training strategies are further introduced to enhance the model performance, yielding more accurate results. Experiments on dataset VIPL-HR show that the proposed CoSTHR achieves better results than the state-of-the-art algorithms.

Depression is a severe psychological condition that affects millions of people worldwide. As depression has received more attention in recent years, it has become imperative to develop automatic methods for detecting depression. Although numerous machine learning methods have been proposed for estimating the levels of depression via audio, visual, and audiovisual emotion sensing, several challenges still exist. For example, it is difficult to extract long-term temporal context information from long sequences of audio and visual data, and it is also difficult to select and fuse useful multi-modal information or features effectively. In addition, how to include other information or tasks to enhance the estimation accuracy is also one of the challenges. In this study, we propose a multi-modal adaptive fusion transformer network for estimating the levels of depression. Transformer-based models have achieved state-of-the-art performance in language understanding and sequence modeling. Thus, the proposed transformer-based network is utilized to extract long-term temporal context information from uni-modal audio and visual data in our work. This is the first transformer-based approach for depression detection. We also propose an adaptive fusion method for adaptively fusing useful multi-modal features. Furthermore, inspired by current multi-task learning work, we also incorporate an auxiliary task (depression classification) to enhance the main task of depression level regression (estimation). The effectiveness of the proposed method has been validated on a public dataset (AVEC 2019 Detecting Depression with AI Sub-challenge) in terms of the PHQ-8 scores. Experimental results indicate that the proposed method achieves better performance compared with currently state-of-the-art methods. Our proposed method achieves a concordance correlation coefficient (CCC) of 0.733 on AVEC 2019 which is 6.2% higher than the accuracy (CCC = 0.696) of the state-of-the-art method.