LLM-Enhanced Multi-Teacher Knowledge Distillation for Modality-Incomplete Emotion Recognition in Daily Healthcare
- 30 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Biomedical and Health Informatics
- Vol. 29 (9) , 6406-6416
Abstract
The critical importance of monitoring and recognizing human emotional states in healthcare has led to a surge in proposals for EEG-based multimodal emotion recognition in recent years. However, practical challenges arise in acquiring EEG signals in daily healthcare settings due to stringent data acquisition conditions, resulting in the issue of incomplete modalities. Existing studies have turned to knowledge distillation as a means to mitigate this problem by transferring knowledge from multimodal networks to unimodal ones. However, these methods are constrained by the use of a single teacher model to transfer integrated feature extraction knowledge, particularly concerning spatial and temporal features in EEG data. To address this limitation, we propose a multi-teacher knowledge distillation framework enhanced with a Large Language Model (LLM), aimed at facilitating effective feature learning in the student network by transferring knowledge of extracting integrated features. Specifically, we employ an LLM as the teacher for extracting temporal features and a graph convolutional neural network for extracting spatial features. To further enhance knowledge distillation, we introduce causal masking and a confidence indicator into the LLM to facilitate the transfer of the most discriminative features. Extensive testing on the DEAP and MAHNOB-HCI datasets demonstrates that our model outperforms existing methods in the modality-incomplete scenario. This study underscores the potential application of large models in this field. The code is publicly available at https://github.com/yuzhezhangEEG/LM-KD .Keywords
Funding Information
- National Natural Science Foundation of China (62192781, 62202367)
- Project of China Knowledge Centre for Engineering Science and Technology
- Project of Chinese Academy of Engineering
- Researchers Supporting Project, King Saud University Riyadh (RSPD2024R968)
This publication has 35 references indexed in Scilit:
- Cross-modal credibility modelling for EEG-based multimodal emotion recognitionJournal of Neural Engineering, 2024
- AutoEER: automatic EEG-based emotion recognition with neural architecture searchJournal of Neural Engineering, 2023
- BioSignal Copilot: Leveraging the power of LLMs in drafting reports for biomedical signalsmedRxiv, 2023
- Distilling EEG representations via capsules for affective computingPattern Recognition Letters, 2023
- Multimodal Emotion Recognition Using CNN-SVM with Data Augmentation2022
- Visual-to-EEG cross-modal knowledge distillation for continuous emotion recognitionPattern Recognition, 2022
- Deep learning with convolutional neural networks for EEG-based music emotion decoding and visualizationBrain-Apparatus Communication: A Journal of Bacomics, 2022
- Confidence-Aware Multi-Teacher Knowledge Distillation2022
- Emotion recognition using multimodal deep learning in multiple psychophysiological signals and videoInternational Journal of Machine Learning and Cybernetics, 2020
- Spectral Graph Theory1996