Measuring Knowledge Structure: Reliability of Concept Mapping Assessment in Medical Education
Authors:
- Matthew McElvanyMMMatthew McElvany
- Jane M. ShayJSJane M. Shay
- Richard J. ShavelsonRSRichard J. Shavelson
- Daniel C. WestDWDaniel C. West
- 1 December 2008
- journal article
- research article
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 83 (12) , 1196-1203
Abstract
To test the reliability of concept map assessment, which can be used to assess an individual's "knowledge structure," in a medical education setting. In 2004, 52 senior residents (pediatrics and internal medicine) and fourth-year medical students at the University of California-Davis School of Medicine created separate concept maps about two different subject domains (asthma and diabetes) on two separate occasions each (four total maps). Maps were rated using four different scoring systems: structural (S; counting propositions), quality (Q; rating the quality of propositions), importance/quality (I/Q; rating importance and quality of propositions), and a hybrid system (H; combining elements of S with I/Q). The authors used generalizability theory to determine reliability. Learners (universe score) contributed 40% to 44% to total score variation for the Q, I/Q, and H scoring systems, but only 10% for the S scoring system. There was a large learner-occasion-domain interaction effect (19%-23%). Subsequent analysis of each subject domain separately demonstrated a large learner-occasion interaction effect (31%-37%) and determined that administration on four to five occasions was necessary to achieve adequate reliability. Rater variation was uniformly low. The Q, I/Q, and H scoring systems demonstrated similar reliability and were all more reliable than the S system. The findings suggest that training and practice are required to perform the assessment task, and, as administered in this study, four to five testing occasions are required to achieve adequate reliability. Further research should focus on whether alterations in the concept mapping task could allow it to be administered over fewer occasions while maintaining adequate reliability.Keywords
This publication has 10 references indexed in Scilit:
- Windows into the mindHigher Education, 2005
- Comparison of two concept-mapping techniques: Implications for scoring, interpretation, and useJournal of Research in Science Teaching, 2005
- Concept mapping assessment in medical education: a comparison of two scoring systemsMedical Education, 2002
- On the Validity of Cognitive Interpretations of Scores From Alternative Concept-Mapping TechniquesEducational Assessment, 2001
- Comparison of the reliability and validity of scores from two concept-mapping techniquesJournal of Research in Science Teaching, 2001
- Critical Thinking in Graduate Medical EducationJAMA, 2000
- Concept map assessment of classroom learning: Reliability, validity, and logistical practicalityJournal of Research in Science Teaching, 1999
- Using concept maps to assess student learning in the science classroom: Must different methods compete?Journal of Research in Science Teaching, 1998
- Problems and issues in the use of concept maps in science assessmentJournal of Research in Science Teaching, 1996
- Concept mapping: A useful tool for science educationJournal of Research in Science Teaching, 1990