Standard setting: Comparison of two methods
Open Access
- 14 September 2006
- journal article
- research article
- Published by Springer Nature in BMC Medical Education
- Vol. 6 (1) , 46
Abstract
The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method. The norm – reference method of standard -setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method. The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% – 87%). The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59–0.74. There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability.Keywords
This publication has 24 references indexed in Scilit:
- Setting Standards in a Professional Higher Education Course: Defining the Concept of the Minimally Competent Student in Performance‐Based Assessment at the Level of Graduation from Medical SchoolHigher Education Quarterly, 2006
- A Model for Setting Performance Standards for Standardized Patient ExaminationsEvaluation & the Health Professions, 2003
- How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory StudyEducational and Psychological Measurement, 1999
- Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judgesMedical Education, 1999
- Standard Setting: An Alternative ApproachJournal of Educational Measurement, 1997
- The Credibility and Comparability of StandardsApplied Measurement in Education, 1997
- Standard setting in medical educationAcademic Medicine, 1996
- Crossing the cultural doctor-patient barrierAcademic Medicine, 1996
- Validating the Performance Standards Associated With Passing ScoresReview of Educational Research, 1994
- The Angoff Cutoff Score Method: The Impact of Frame-of-Reference Rater TrainingEducational and Psychological Measurement, 1991