Interpretable MultiModal AI for Mental Health Detection via Facial and Vocal Emotions
Open Access
- 23 January 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 17-22
Abstract
Mental disorders, including depression and anxiety, constitute a major global health concern. Timely and precise identification is essential for effective treatment and suicide prevention. Human emotions, conveyed through facial expressions and vocalizations, provide an essential means for identification. However, current Artificial Intelligence (AI) frameworks in this domain face two major limitations: dataset imbalance and resultant low diagnostic accuracy. Moreover, previous work considered only one modality. This paper presents a comprehensive, multi-modal approach using facial and vocal emotions to improve mental health identification. A Generative Adversarial Network (GAN) is employed to mitigate data scarcity and imbalance by generating a variety of high-quality data samples, utilizing the public facial (FER2013) and vocal (CREMA-D) emotion datasets for training. Following data balancing, the YOLOv8 model is employed to generate a dataset categorized into specific mental disorder classifications (e.g., depression, anxiety, another disorder, and no disorder). A comparative analysis of various Convolutional Neural Network (CNN) architectures is performed to identify the most efficient classification model based on accuracy. The best model is subsequently analyzed using GRAD-CAM (Gradient-weighted Class Activation Mapping) to enhance the interpretability of its internal decision-making process. For real-world implementation and validation, a secure, full-stack web application is developed. This application uses streamlit for front-end, flask server for hosting CNN model (back-end), HTTP for communication and JSON web token for data security. The framework is tested in real-time using live facial and voice data collected from Sunway University students. This research establishes a high-accuracy framework for mental health recognition from multi-modal affective inputs by utilizing Generative Adversarial Networks (GANs) for effective data augmentation and integrating sophisticated Convolutional Neural Network (CNN) and YOLOv8 methodologies. The developed application facilitates accurate and expeditious detection, thereby enabling early intervention and the mitigation of adverse mental health outcomes.Keywords
This publication has 17 references indexed in Scilit:
- Enhancing Breast Cancer Diagnosis in Mammography: Evaluation and Integration of Convolutional Neural Networks and Explainable AI2024
- An explainable fast deep neural network for emotion recognitionBiomedical Signal Processing and Control, 2024
- Bias in medical AI: Implications for clinical decision-makingPLOS Digital Health, 2024
- A Hybrid Learning-Architecture for Mental Disorder Detection Using Emotion RecognitionIEEE Access, 2024
- Early detection of depression through facial expression recognition and electroencephalogram-based artificial intelligence-assisted graphical user interfaceNeural Computing and Applications, 2024
- Artificial intelligence and assistive technology: risks, rewards, challenges, and opportunitiesAssistive Technology, 2023
- SMG: A Micro-gesture Dataset Towards Spontaneous Body Gestures for Emotional Stress State AnalysisInternational Journal of Computer Vision, 2023
- REDE - Detecting human emotions using CNN and RASA2022
- Recognition and Visualization of Facial Expression and Emotion in Healthcare2021
- The persuasive power of emotions: Effects of emotional expressions on attitude formation and change.Journal of Applied Psychology, 2015