MMKT: Multimodal Knowledge Tracing in Personalized E-Learning Systems for Supporting Lifelong Learning
- 19 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. 12 (6) , 5179-5198
Abstract
To effectively support lifelong learning, personalized e-learning systems aim to provide customized educational services tailored to individual learners’ needs. A crucial aspect for enabling these intelligent services is knowledge tracing, which involves dynamically modeling learners’ knowledge acquisition based on their past learning records. Traditional deep knowledge tracing models primarily focus on learners’ responses and the knowledge components covered in exercises, often neglecting other valuable multimodal learning features. In response to this limitation, our contribution is the introduction of the multimodal knowledge tracing (MMKT) model, which leverages four distinct learning modalities: textual, image, cognitive parametric, and knowledge association modalities. MMKT initially employs a combination of pretrained models and fine-tuning to obtain representations for text and image modalities. Using the EM algorithm, it acquires the cognitive parameter modality for exercises and learners. Subsequently, a multimodal fusion is applied to obtain a comprehensive exercise representation. In the next step, the model explores knowledge impact weights through the knowledge association modality. This information is combined with the cognitive parameter modality of the learner and the hidden states tracked by the gated recurrent unit, resulting in a multimodal representation of the learner’s knowledge state. Experiments conducted on publicly available datasets demonstrate the significant improvement of the proposed MMKT model. The proposed model outperforms the widely used baselines, deep knowledge tracing and dynamic key–value memory networks, achieving relative AUC gains of 3.57% and 6.37%, respectively, in predicting student learning performance. This study provides valuable insights that assist learners in understanding their knowledge mastery and contribute to the realization of lifelong learning from a technical standpoint.Keywords
Funding Information
- National Key R&D Program of China (2023YFC3305704)
- National Natural Science Foundation of China (62177022)
- National Natural Science Foundation of China (61901165)
- National Natural Science Foundation of China (61501199)
- Youth Al Talents Fund of the Chinese Association of Automation under Major Program (HBRC-JKYZD-2024-310)
- Self-Determined Research Funds of CCNU From the Colleges’ Basic Research and Operation of MOE (CCNU25ai011)
- Self-Determined Research Funds of CCNU From the Colleges’ Basic Research and Operation of MOE (CCNU25ai039)
This publication has 49 references indexed in Scilit:
- Improving the performance and explainability of knowledge tracing via Markov blanketInformation Processing & Management, 2024
- Heterogeneous graph-based knowledge tracing with spatiotemporal evolutionExpert Systems with Applications, 2023
- Response speed enhanced fine-grained knowledge tracing: A multi-task learning perspectiveExpert Systems with Applications, 2023
- A deep cross-modal neural cognitive diagnosis framework for modeling student performanceExpert Systems with Applications, 2023
- Lifelong Learning in the Educational Setting: A Systematic Literature ReviewThe Asia-Pacific Education Researcher, 2023
- Research trends in multimodal learning analytics: A systematic mapping studyComputers and Education: Artificial Intelligence, 2023
- A survey on deep learning based knowledge tracingKnowledge-Based Systems, 2022
- Feature-level fusion approaches based on multimodal EEG data for depression recognitionInformation Fusion, 2020
- Higher-Order Latent Trait Models for Cognitive DiagnosisPsychometrika, 2004
- Knowledge tracing: Modeling the acquisition of procedural knowledgeUser Modeling and User-Adapted Interaction, 1995