Abstract
To effectively support lifelong learning, personalized e-learning systems aim to provide customized educational services tailored to individual learners’ needs. A crucial aspect for enabling these intelligent services is knowledge tracing, which involves dynamically modeling learners’ knowledge acquisition based on their past learning records. Traditional deep knowledge tracing models primarily focus on learners’ responses and the knowledge components covered in exercises, often neglecting other valuable multimodal learning features. In response to this limitation, our contribution is the introduction of the multimodal knowledge tracing (MMKT) model, which leverages four distinct learning modalities: textual, image, cognitive parametric, and knowledge association modalities. MMKT initially employs a combination of pretrained models and fine-tuning to obtain representations for text and image modalities. Using the EM algorithm, it acquires the cognitive parameter modality for exercises and learners. Subsequently, a multimodal fusion is applied to obtain a comprehensive exercise representation. In the next step, the model explores knowledge impact weights through the knowledge association modality. This information is combined with the cognitive parameter modality of the learner and the hidden states tracked by the gated recurrent unit, resulting in a multimodal representation of the learner’s knowledge state. Experiments conducted on publicly available datasets demonstrate the significant improvement of the proposed MMKT model. The proposed model outperforms the widely used baselines, deep knowledge tracing and dynamic key–value memory networks, achieving relative AUC gains of 3.57% and 6.37%, respectively, in predicting student learning performance. This study provides valuable insights that assist learners in understanding their knowledge mastery and contribute to the realization of lifelong learning from a technical standpoint.
Funding Information
  • National Key R&D Program of China (2023YFC3305704)
  • National Natural Science Foundation of China (62177022)
  • National Natural Science Foundation of China (61901165)
  • National Natural Science Foundation of China (61501199)
  • Youth Al Talents Fund of the Chinese Association of Automation under Major Program (HBRC-JKYZD-2024-310)
  • Self-Determined Research Funds of CCNU From the Colleges’ Basic Research and Operation of MOE (CCNU25ai011)
  • Self-Determined Research Funds of CCNU From the Colleges’ Basic Research and Operation of MOE (CCNU25ai039)

This publication has 49 references indexed in Scilit: