Lucky guess or knowledge: a cross-sectional study using the Bland and Altman analysis to compare confidence-based testing of pharmacological knowledge in 3rd and 5th year medical students
- 8 August 2014
- journal article
- research article
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 20 (2) , 431-440
Abstract
Multiple-choice-questions are common in medical examinations, but guessing biases assessment results. Confidence-based-testing (CBT) integrates indicated confidence levels. It has been suggested that correctness of and confidence in an answer together indicate knowledge levels thus determining the quality of a resulting decision. We used a CBT approach to investigate whether decision quality improves during undergraduate medical education. 3rd- and 5th-year students attended formative multiple-choice exams on pharmacological issues. Students were asked to indicate their confidence in a given answer. Correctness of answers was scored binary (1-correct; 0-wrong) and confidence levels were transformed to an ordinal scale (guess: 0; rather unsure: 0.33; rather sure: 0.66; very sure: 1). 5th-year students gave more correct answers (73 ± 16 vs. 49 ± 13 %, p < 0.05) and were on average more confident regarding the correctness of their answers (0.61 ± 0.18 vs. 0.46 ± 0.13, p < 0.05). Correlation of these parameters was stronger for 5th-year students (r = 0.81 vs. r = 0.52), but agreement of confidence and correctness (‘centration’) was lower. By combining the Bland-and-Altman approach with categories of decision-quality we found that 5th-year students were more likely to be ‘well-informed’ (41 vs. 5 %), while more 3rd-students were ‘uninformed’ (24 vs. 76 %). Despite a good correlation of exam results and confidence in given answers increased knowledge might be accompanied by a more critical view at the own abilities. Combining the statistical Bland-and-Altman analysis with a theoretical approach to decision-quality, more advanced students are expected to apply correct beliefs, while their younger fellows are rather at risk to hesitate or to act amiss.Keywords
This publication has 15 references indexed in Scilit:
- Foundation year 1 doctors and clinical pharmacology and therapeutics teaching. A retrospective view in light of experienceBritish Journal of Clinical Pharmacology, 2007
- Quality assurance of multiple‐choice testsQuality Assurance in Education, 2006
- Medical Teacher 25th Anniversary SeriesMultiple-choice questions revisitedMedical Teacher, 2004
- A New Look at the Influence of Guessing on the Reliability of Multiple-Choice TestsApplied Psychological Measurement, 2003
- The concept of knowledge and how to measure itJournal of Intellectual Capital, 2003
- Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments.Journal of Personality and Social Psychology, 1999
- An international comparison of knowledge levels of medical students: the Maastricht Progress TestMedical Education, 1996
- The assessment of clinical skills/competence/performanceAcademic Medicine, 1990
- STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENTThe Lancet, 1986
- Anomalous Responses on Confidence-Scored TestsEvaluation & the Health Professions, 1985