TCMI-F-6D benchmark construction and quantitative assessment of interdisciplinary foundational competencies in traditional Chinese medicine informatics using large language models
Authors:
Open Access
- 7 April 2026
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Artificial Intelligence
- Vol. 9, 1780967
Abstract
Introduction: Traditional Chinese Medicine Informatics (TCMI), as an emerging interdisciplinary field, places high demands on foundational interdisciplinary competency assessment in its talent cultivation and research practices. However, Large Language Models (LLMs) currently lack a suitable quantitative assessment system tailored to the characteristics of TCMI. Methods: To address this gap, this study, grounded in Cognitive Hierarchy Theory and Disciplinary Knowledge Structure Theory, selected six core disciplines closely related to TCMI from the Massive Multitask Language Understanding (MMLU) dataset, constructed an evaluation framework for foundational interdisciplinary competency in TCMI-related scenarios, and established the TCMI-F-6D (the TCMI-Foundation-6 Domain Benchmark) together with a composite metric system. Three experiments were conducted to evaluate the models’ baseline capability, learning gains, and performance stability. The experiments comprehensively assessed the competency of 20 LLMs across 8 categories, and selected 6 models with weaker overall performance for focused analysis of their interdisciplinary competency characteristics. Results: The results showed that, among the base models, ChatGLM3-6B performed best in interdisciplinary knowledge integration (43.97%), while DeepSeek-V3.1 achieved the best overall application performance (80.87%) among the chat models. Specifically, Qwen-14B-Chat also demonstrated stable and predictable learning performance under varying example conditions, with an average learning gain of 5.60% and a 95% confidence interval (CI) of [5.50%, 5.70%]. Discussion: Collectively, this study clarifies the differences in foundational interdisciplinary competency among LLMs in this discipline, providing a quantifiable assessment framework, methodological support, and empirical evidence for TCMI’s educational, research tool selection, and the implementation of a standardized interdisciplinary competency assessment system.Keywords
Funding Information
- National Science and Technology Major Project (2024ZD0532401)
- Major Scientific Research Project of Universities in Anhui Province (2024AH040143)
- Teaching Research Projects of Anhui University of Chinese Medicine (2024xjjy_zdzx005)
- Teaching Quality Engineering Projects for Higher Education in Anhui Province (2023sdxx026)
- Postgraduate Education Quality Engineering Projects of Anhui Province (2024cxcysj125)
- Postgraduate Education Quality Engineering Projects of Anhui Province (2024qygzz028)
- Postgraduate Education Quality Engineering Projects of Anhui Province (2024jyjxggyjY215)
This publication has 15 references indexed in Scilit:
- Bloom Meets Gen AI: Reconceptualising Bloom's Taxonomy in the Era of Co-piloted LearningPreprints.org, 2025
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark2025
- Utilizing large language models for identifying future research opportunities in environmental scienceJournal of Environmental Management, 2024
- Incorporating Bloom’s taxonomy into promoting cognitive thinking mechanism in artificial intelligence-supported learning environmentsInteractive Learning Environments, 2024
- Surveying the landscape of environmental social science: a bibliometric and network analysisEnvironmental Research Communications, 2024
- Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive BenchmarkmedRxiv, 2024
- CMMLU: Measuring massive multitask language understanding in Chinese2024
- Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models2024
- Large AI Models in Health Informatics: Applications, Challenges, and the FutureIEEE Journal of Biomedical and Health Informatics, 2023
- The blind spots of interdisciplinarity in addressing grand challengesCritical Perspectives on Accounting, 2022