Abstract
This study introduces an innovative cross‐modal coattention network (CMCAN) framework, specifically designed to tackle the challenges of temporal misalignment and modality‐specific feature preservation in multimodal depression detection. The architecture comprises two fundamental components: (1) guided multihead attention, which facilitates context‐aware interactions across modalities, and (2) dedicated self‐attention pathways that ensure the preservation of key unimodal features. To enhance the estimation of depression severity, a cascaded fusion strategy is employed, combining feature superposition with hierarchical stacking. When evaluated on a Chinese localized depression dataset, CMCAN demonstrates exceptional performance, achieving optimal results (accuracy: 0.839; precision: 0.829; recall: 0.863) with its audiovisual guided coattention module (AVGA (SA (V), SA (A))) comprising three cascaded layers. The framework consistently surpasses unimodal baselines across various emotional valence stimuli (positive, neutral, and negative) and achieves state‐of‐the‐art performance on AVEC 2014 (MAE: 5.38). Comprehensive ablation studies validate the effectiveness of individual components, whereas comparative analyses demonstrate significant improvements over existing multimodal fusion approaches. These findings underscore the robustness and generalizability of CMCAN, validating its effectiveness in harmonizing cross‐modal synergy while preserving modality‐specific features, thereby advancing practical solutions for automated depression detection.
Funding Information
  • National Natural Science Foundation of China (62267008)
  • National Natural Science Foundation of China (62067008)

This publication has 60 references indexed in Scilit: