Xiaokang Yu

In recent years, substantial research has focused on automated systems for assessing depression levels using different types of data, such as audio and visual inputs. However, signals recorded from individuals with depression can be influenced by external factors, such as the recording equipment and environment, making it essential to create a system that is resilient to these interferences to maintain accuracy. This study introduces a fused-attention model for evaluating depression severity using enhanced multi-modal data inputs. Applying several pre-trained advanced models, this article incorporates audiovisual sequences with augmentation. The framework includes two novel components, which we termed as the FIE and VIE blocks, for extracting detailed facial and vocal features. The FIE block utilizes ResNet-18 to enhance the feature representation of video frames and integrates two types of attention mechanisms to capture spatial-temporal patterns. Meanwhile, the VIE block processes the Mel spectrogram of the audio signal, followed by an optimized Swin transformer block to extract auditory features. The model demonstrates strong performance, accurately identifying depression severity in 3-second audiovisual sequences with an 81.4% accuracy rate on the AVEC2014 dataset, and achieves a Kappa score of 0.731 and an MF1 index of 0.798. Furthermore, it shows high resilience to noise, underscoring its ability to mitigate the effects of recording equipment and environmental conditions in depression level estimation.

Single-channel EEG based sleep staging is of interest to researchers due to its broad application prospect in daily sleep monitoring recently. We proposed using contextual scalograms as input and developed a convolutional neural network with attention modules named Co-ScaleNet for sleep staging. The contextual scalograms were obtained by combining the same color channels of three original RGB scalograms from consecutive epochs, and a simple and efficient data augmentation was designed according to their various forms. The Co-ScaleNet consists of two main parts. Firstly, three parallel convolutional branches with attention modules correspondingly extract and fuse features from contextual scalograms at the top layers. The remaining part is a stack of lightweight blocks. We achieved an overall accuracy of 87.0% for healthy individuals, 84.7% for depressed patients. And we obtained comparable performance on the public Sleep-EDFx (82.8%), ISRUC (84.6%) and SHHS datasets (87.7%), including a high recall of N1. The contextual scalograms of R channel as input achieved the best performance, which conform to the features of interest in visual scoring. The attention modules improved the recall of N1 and N3. Overall, the contextual scalograms provided a novel scheme for both contextual information extraction and data augmentation. Our study successfully expanded its application to depression datasets, as well as patients with sleep apnea, demonstrating its wide applicability.