Abstract
Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.
Funding Information
  • National Key Research and Development Program of China (2019YFA0706200)
  • National Natural Science Foundation of China (62227807)
  • Natural Science Foundation of Gansu Province (22JR5RA488)
  • Fundamental Research Funds for the Central Universities (lzujbky-2023-16)
  • Supercomputing Center of Lanzhou University

This publication has 68 references indexed in Scilit: