Multi-Modal Fused-Attention Network for Depression Level Recognition Based on Enhanced Audiovisual Cues

Authors:
Open Access
Publisher website
Google Scholar
Abstract
In recent years, substantial research has focused on automated systems for assessing depression levels using different types of data, such as audio and visual inputs. However, signals recorded from individuals with depression can be influenced by external factors, such as the recording equipment and environment, making it essential to create a system that is resilient to these interferences to maintain accuracy. This study introduces a fused-attention model for evaluating depression severity using enhanced multi-modal data inputs. Applying several pre-trained advanced models, this article incorporates audiovisual sequences with augmentation. The framework includes two novel components, which we termed as the FIE and VIE blocks, for extracting detailed facial and vocal features. The FIE block utilizes ResNet-18 to enhance the feature representation of video frames and integrates two types of attention mechanisms to capture spatial-temporal patterns. Meanwhile, the VIE block processes the Mel spectrogram of the audio signal, followed by an optimized Swin transformer block to extract auditory features. The model demonstrates strong performance, accurately identifying depression severity in 3-second audiovisual sequences with an 81.4% accuracy rate on the AVEC2014 dataset, and achieves a Kappa score of 0.731 and an MF1 index of 0.798. Furthermore, it shows high resilience to noise, underscoring its ability to mitigate the effects of recording equipment and environmental conditions in depression level estimation.
Funding Information
  • Basic and Applied Basic Research Foundation of Guangdong Province (2023A1515011824)
  • Shenzhen Municipal Fundamental Research Program (JCYJ20240813151223031)

This publication has 55 references indexed in Scilit: