Multimodal Depression Detection Using Audio Visual Cues
Authors:
- 10 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Millions of people all around the world suffer from depression, which is a common mental health issue. Effective intervention and therapy depend on early and precise identification of depression. Multimodal techniques that incorporate audio and video data have recently demonstrated promising outcomes in the detection of depression. In this article, authors suggest a convolutional neural network (CNN) model for multimodal depression identification based on audio and video. The proposed model makes use of audio and visual elements to gather comprehensive and complementary data about depression. Pitch, intensity, and spectral information, among others, are derived from voice recordings for the audio modality. The video modality uses face landmarks, optical flow, and pose estimation techniques to combine facial expressions, head movements, and body gestures. The parallel branches of the CNN architecture process the audio and visual inputs individually. To learn discriminative audio representations, the audio branch applies three convolutional layers, followed by pooling and dense layers. To extract video-specific information, the video branch uses five convolutional layers with different filter widths and depths, followed by fully connected and pooling layers. Further a late fusion technique has been adopted for multimodal fusion, concatenating learned features from the two modalities and passing them through additional thick layers for depression prediction. Authors also used regularization strategies, including dropout and batch normalization, during training to alleviate the overfitting problem. Authors evaluated the proposed multimodal CNN model on a DAIC-WOZ dataset consisting of audio and video recordings of individuals with and without depression. The proposed model achieved 77% accuracy and hence demonstrate that the proposed multimodal CNN model achieves superior performance compared to unimodal approaches.Keywords
This publication has 20 references indexed in Scilit:
- Review on Automated Depression Detection from audio visual clue using Sentiment Analysis2021
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- Multi-modality Depression Detection via Multi-scale Temporal Dilated CNNs2019
- A Multi-Modal Hierarchical Recurrent Neural Network for Depression Detection2019
- Novel Audio Features for Music Emotion RecognitionIEEE Transactions on Affective Computing, 2018
- Topic Modeling Based Multi-modal Depression Detection2017
- Multimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features2016
- Automated audiovisual depression analysisCurrent Opinion in Psychology, 2015
- Depression Estimation Using Audiovisual Features and Fisher Vector Encoding2014
- Model Fusion for Multimodal Depression Classification and Level Detection2014