A Transformer‐Based Cross‐Modal Coattention Framework for Multimodal Depression Detection
Open Access
- 14 January 2026
- journal article
- Published by Wiley in International Journal of Intelligent Systems
- Vol. 2026 (1)
Abstract
This study introduces an innovative cross‐modal coattention network (CMCAN) framework, specifically designed to tackle the challenges of temporal misalignment and modality‐specific feature preservation in multimodal depression detection. The architecture comprises two fundamental components: (1) guided multihead attention, which facilitates context‐aware interactions across modalities, and (2) dedicated self‐attention pathways that ensure the preservation of key unimodal features. To enhance the estimation of depression severity, a cascaded fusion strategy is employed, combining feature superposition with hierarchical stacking. When evaluated on a Chinese localized depression dataset, CMCAN demonstrates exceptional performance, achieving optimal results (accuracy: 0.839; precision: 0.829; recall: 0.863) with its audiovisual guided coattention module (AVGA (SA (V), SA (A))) comprising three cascaded layers. The framework consistently surpasses unimodal baselines across various emotional valence stimuli (positive, neutral, and negative) and achieves state‐of‐the‐art performance on AVEC 2014 (MAE: 5.38). Comprehensive ablation studies validate the effectiveness of individual components, whereas comparative analyses demonstrate significant improvements over existing multimodal fusion approaches. These findings underscore the robustness and generalizability of CMCAN, validating its effectiveness in harmonizing cross‐modal synergy while preserving modality‐specific features, thereby advancing practical solutions for automated depression detection.Keywords
Funding Information
- National Natural Science Foundation of China (62267008)
- National Natural Science Foundation of China (62067008)
This publication has 60 references indexed in Scilit:
- LMTformer: facial depression recognition with lightweight multi-scale transformer from videosApplied Intelligence, 2024
- Depression Diagnosis and Analysis via Multimodal Multi-order Factor Fusion2024
- Depression recognition base on acoustic speech model of Multi-task emotional stimulusBiomedical Signal Processing and Control, 2023
- PRA-Net: Part-and-Relation Attention Network for depression recognition from facial expressionComputers in Biology and Medicine, 2023
- A multimodal fusion model with multi-level attention mechanism for depression detectionBiomedical Signal Processing and Control, 2022
- Deep learning for depression recognition with audiovisual cues: A reviewInformation Fusion, 2022
- Causes and Consequences of Diagnostic Heterogeneity in Depression: Paths to Discovering Novel Biological Depression SubtypesBiological Psychiatry, 2020
- A Sociological Analysis of Depression in China2020
- Ambient Sound Provides Supervision for Visual Learning2016
- Communicating Without Words2010