Abstract
Mental disorders, including depression and anxiety, constitute a major global health concern. Timely and precise identification is essential for effective treatment and suicide prevention. Human emotions, conveyed through facial expressions and vocalizations, provide an essential means for identification. However, current Artificial Intelligence (AI) frameworks in this domain face two major limitations: dataset imbalance and resultant low diagnostic accuracy. Moreover, previous work considered only one modality. This paper presents a comprehensive, multi-modal approach using facial and vocal emotions to improve mental health identification. A Generative Adversarial Network (GAN) is employed to mitigate data scarcity and imbalance by generating a variety of high-quality data samples, utilizing the public facial (FER2013) and vocal (CREMA-D) emotion datasets for training. Following data balancing, the YOLOv8 model is employed to generate a dataset categorized into specific mental disorder classifications (e.g., depression, anxiety, another disorder, and no disorder). A comparative analysis of various Convolutional Neural Network (CNN) architectures is performed to identify the most efficient classification model based on accuracy. The best model is subsequently analyzed using GRAD-CAM (Gradient-weighted Class Activation Mapping) to enhance the interpretability of its internal decision-making process. For real-world implementation and validation, a secure, full-stack web application is developed. This application uses streamlit for front-end, flask server for hosting CNN model (back-end), HTTP for communication and JSON web token for data security. The framework is tested in real-time using live facial and voice data collected from Sunway University students. This research establishes a high-accuracy framework for mental health recognition from multi-modal affective inputs by utilizing Generative Adversarial Networks (GANs) for effective data augmentation and integrating sophisticated Convolutional Neural Network (CNN) and YOLOv8 methodologies. The developed application facilitates accurate and expeditious detection, thereby enabling early intervention and the mitigation of adverse mental health outcomes.

This publication has 17 references indexed in Scilit: