Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education – a retrospective, single-center, mixed model analysis of progress testing results
Open Access
- 1 January 2021
- journal article
- research article
- Published by Taylor & Francis in Medical Education Online
- Vol. 26 (1) , 1972505
Abstract
Medical education research focuses on the development of efficient learning methods promoting the acquisition of student’s knowledge and competencies. Evaluation of any modification of educational approaches needs to be evaluated accordingly and a reliable effect size needs to be reached. Our aim is to provide a methodological basis to calculate effect sizes from longitudinal progress test data that can be used as reference values in further research. We used longitudinally collected progress test data and evaluated the increasing knowledge of medical students from the first to the fifth academic year. Students were asked to participate in the progress test, which consists of 200 multiple-choice questions in single best answer format with an additional ‘don’t know’ option. All available individual test scores of all progress tests (n = 10) administered between April 2012 and October 2017 were analyzed. Due to the large amount of missing test results, e.g., from students at the beginning of their studies, a linear mixed model was fitted to include all collected data. In total, we analyzed 6324 test scores provided by 2587 medical students. Mean score for medical knowledge (% correct answers) increases from 16.6% (SD: 10.8%) to 51.0% (SD: 15.7%, overall effects size using linear mixed models d = 1.55). Medical students showed a learning effect of d = 0.54 (total gain: 6.9%) between the 1st and 2nd, d = 0.88 (total gain: 12.0%) between the 2nd and 3rd, d = 0.60 (total gain: 7.9%) between the 3rd and 4th and d = 0.58 (total gain: 7.9%) between the 4th and 5th study year. We demonstrated that incomplete data from longitudinally collected progress tests can be used to acquire reliable effect size estimates. The demonstrated effects size between d = 0.53–0.9 by study year may help researchers to design studies in medical education.Keywords
This publication has 30 references indexed in Scilit:
- Development of knowledge in basic sciences: a comparison of two medical curriculaMedical Education, 2012
- If you teach them, they will learn: why medical education needs comparative effectiveness researchAdvances in Health Sciences Education, 2012
- The use of progress testingPerspectives on Medical Education, 2012
- Does Simulation-Based Medical Education With Deliberate Practice Yield Better Results Than Traditional Clinical Education? A Meta-Analytic Comparative Review of the EvidenceAcademic Medicine, 2011
- Computerized Virtual Patients in Health Professions Education: A Systematic Review and Meta-AnalysisAcademic Medicine, 2010
- Impact on knowledge acquisition of the transition from a conventional to an integrated contextual medical curriculumMedical Education, 2009
- Evaluating the effectiveness of curriculum change. Is there a difference between graduating student outcomes from two different curricula?Medical Teacher, 2009
- Benchmarking by cross‐institutional comparison of student achievement in a progress testMedical Education, 2007
- The quarterly profile examinationAcademic Medicine, 1990
- The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One TutoringEducational Researcher, 1984