M&M: Multimodal-Multitask Model Integrating Audiovisual Cues in Cognitive Load Assessment
Preprint
Abstract
This paper introduces the M&M model, a novel multimodal-multitask learning framework, applied to the AVCAffe dataset for cognitive load assessment (CLA). M&M uniquely integrates audiovisual cues through a dual-pathway architecture, featuring specialized streams for audio and video inputs. A key innovation lies in its cross-modality multihead attention mechanism, fusing the different modalities for synchronized multitasking. Another notable feature is the model's three specialized branches, each tailored to a specific cognitive load label, enabling nuanced, task-specific analysis. While it shows modest performance compared to the AVCAffe's single-task baseline, M\&M demonstrates a promising framework for integrated multimodal processing. This work paves the way for future enhancements in multimodal-multitask learning systems, emphasizing the fusion of diverse data types for complex task handling.Keywords
All Related Versions
- Version 1, 2024-03-14, ArXiv
- Published version: , 869.
This publication has 24 references indexed in Scilit:
- Bioinspired multisensory neural network with crossmodal integration and recognitionNature Communications, 2021
- Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series dataNeurocomputing, 2020
- Multi-channel spectrograms for speech processing applications using deep learning methodsPattern Analysis and Applications, 2020
- Towards measuring cognitive load through multimodal physiological dataCognition, Technology & Work, 2020
- Deep Convolutional Neural Networks for mental load classification based on EEG dataPattern Recognition, 2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Fundamental Cognitive Workload Assessment: A Machine Learning Comparative Approach2017
- Measuring Cognitive Load Using Linguistic Features: Implications for Usability Evaluation and Adaptive Interaction DesignInternational Journal of Human–Computer Interaction, 2014
- How cognitive load affects duration judgments: A meta-analytic reviewActa Psychologica, 2010
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008