Secure AI-assisted angoff standard-setting for single best answer questions: A non-inferiority validation study
Authors:
Open Access
- 11 June 2026
- journal article
- Published by Taylor & Francis in Medical Teacher
- p. 1-11
Abstract
The Angoff method sets exam pass scores using expert estimates of the probability that a borderline candidate will answer each question correctly; this process is time-intensive and variable. We evaluated whether AI-derived Angoff estimates are non-inferior to human judgements for single best answer (SBA) questions using a secure offline question feature-extraction tool. We conducted a methodological study in a UK Physician Associate programme. A faculty survey informed a borderline Year-2 student descriptor. ExamFeats, a secure offline tool, extracted surface features from 1,003 historical and 100 new SBAs. Three AI models were evaluated: (A) a large language model (LLM) using extracted features and the borderline-student descriptor; (B) a machine-learning model (ML) using ridge regression and (C) a hybrid combining LLM and ML predictions. Human Angoff ratings for the 100 new items were averaged across 3-4 faculty members. The primary outcome was the mean difference between AI and human Angoff scores, assessed against a prespecified 10% non-inferiority margin, with additional analysis of discordant items (>10% difference). For the 100 new SBAs, human Angoff estimates and all AI models produced closely aligned distributions, with minimal systematic differences. The human mean Angoff score was 60.3%, while Model A, Model B and Model C produced mean estimates of 60.8%, 60.0% and 60.3%, respectively. There were no significant differences between models (p = 0.41) and 95% confidence-intervals lay within the non-inferiority margin. Discordance between human and model prediction occurred in 33% of SBAs but this was not explained by differences in extracted surface item features. AI-derived Angoff estimates reproduced panel-level Angoff behaviour within the prespecified error bounds and could reduce the time and cognitive burden of standard setting. The secure feature-extraction tool enabled AI-assisted standard setting without sharing item text. However, question-level discordance indicates that AI should augment rather than replace expert panels.Keywords
This publication has 16 references indexed in Scilit:
- Item Difficulty Modeling Using Fine-tuned Small and Large Language ModelsEducational and Psychological Measurement, 2025
- Artificial Intelligence in Medical Education Assessments: Navigating the Challenges to Academic IntegrityMedical Science Educator, 2024
- Rapid Blueprinting: An Efficient Method for Designing Content of AssessmentsMedical Science Educator, 2024
- A checklist for reporting, reading and evaluating Artificial Intelligence Technology Enhanced Learning (AITEL) research in medical educationMedical Teacher, 2024
- Insights into the Angoff method: results from a simulation studyBMC Medical Education, 2016
- Understanding Bland Altman analysisBiochemia Medica, 2015
- Standard setting in medical educationAcademic Medicine, 1996
- Stacked regressionsMachine Learning, 1996
- STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENTThe Lancet, 1986
- A computer readability formula designed for machine scoring.Journal of Applied Psychology, 1975