DepITCM: an audio-visual method for detecting depression
Open Access
- 23 January 2025
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Psychiatry
- Vol. 15, 1466507
Abstract
Introduction: Depression is a prevalent mental disorder, and early screening and treatment are crucial for detecting depression. However, there are still some limitations in the currently proposed deep models based on audio-video data, for example, it is difficult to effectively extract and select useful multimodal information and features from audio-video data, and very few studies have been able to focus on three dimensions of information: time, channel, and space at the same time in depression detection. In addition, there are challenges in utilizing other tasks to enhance prediction accuracy. The resolution of these issues is crucial for constructing models of depression detection.Methods: In this paper, we propose a multi-task representation learning based on vision and audio for depression detection model (DepITCM).The model comprises three main modules: a data preprocessing module, the Inception-Temporal-Channel Principal Component Analysis Module(ITCM Encoder), and a multi-task learning module. To efficiently extract rich feature representations from audio and video data, the ITCM Encoder employs a staged feature extraction strategy, transitioning from global to local features. This approach enables the capture of global features while emphasizing the fusion of temporal, channel, and spatial information in finer detail. Furthermore, inspired by multi-task learning strategies, this paper enhances the primary task of depression classification by incorporating a secondary task (regression task) to improve overall performance.Results: We conducted experiments on the AVEC2017 and AVEC2019 datasets. The results show that, in the classification task, our method achieved an F1 score of 0.823 and a classification accuracy of 0.823 on the AVEC2017 dataset, and an F1 score of 0.816 and a classification accuracy of 0.810 on the AVEC2019 dataset. In the regression task, the RMSE was 6.10 (AVEC2017) and 4.89 (AVEC2019), respectively. These results demonstrate that our method outperforms most existing methods in both classification and regression tasks. Furthermore, we demonstrate that the model proposed in this paper can effectively improve the performance of depression detection when using multi-task learning.Discussion: Although depression detection through multimodality has shown good results in previous studies. However, multi-task learning can utilize the complementary information between different tasks. Therefore, our work combines multimodal and multi-task learning to improve the accuracy of depression detection. Previous studies have mostly focused on the extraction of global features while ignoring the importance of local features. Based on the problems of previous studies, we have made corresponding improvements to provide a more comprehensive and effective solution for depression detection.Keywords
This publication has 31 references indexed in Scilit:
- Channel prior convolutional attention for medical image segmentationComputers in Biology and Medicine, 2024
- Reading Between the Frames: Multi-modal Depression Detection in Videos from Non-verbal Cues2024
- Multimodal Depression Detection Using Audio Visual Cues2023
- An Investigation on the Audio-Video Data Based Estimation of Emotion Regulation Difficulties and Their Association With Mental DisordersIEEE Access, 2023
- A multimodal fusion model with multi-level attention mechanism for depression detectionBiomedical Signal Processing and Control, 2022
- Multimodal temporal machine learning for Bipolar Disorder and Depression RecognitionPattern Analysis and Applications, 2021
- A hierarchical depression detection model based on vocal and emotional cuesNeurocomputing, 2021
- Squeeze-and-Excitation Networks2018
- COVAREP — A collaborative voice analysis repository for speech technologies2014
- The Hamilton Rating Scale for Depression1986