The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value
Open Access
- 29 September 2016
- journal article
- research article
- Published by Springer Nature in BMC Medical Education
- Vol. 16 (1) , 1-10
Abstract
Failure to adhere to standard item-writing guidelines may render examination questions easier or more difficult than intended. Item complexity describes the cognitive skill level required to obtain a correct answer. Higher cognitive examination items promote critical thinking and are recommended to prepare students for clinical training. This study evaluated faculty-authored examinations to determine the impact of item-writing flaws and item complexity on the difficulty and discrimination value of examination items used to assess third year veterinary students. The impact of item-writing flaws and item complexity (cognitive level I-V) on examination item difficulty and discrimination value was evaluated on 1925 examination items prepared by clinical faculty for third year veterinary students. The mean (± SE) percent correct (83.3 % ± 17.5) was consistent with target values in professional education, and the mean discrimination index (0.18 ± 0.17) was slightly lower than recommended (0.20). More than one item-writing flaw was identified in 37.3 % of questions. The most common item-writing flaws were awkward stem structure, implausible distractors, longest response is correct, and responses are series of true-false statements. Higher cognitive skills (complexity level III-IV) were required to correctly answer 38.4 % of examination items. As item complexity increased, item difficulty and discrimination values increased. The probability of writing discriminating, difficult examination items decreased when implausible distractors and all of the above were used, and increased if the distractors were comprised of a series of true/false statements. Items with four distractors were not more difficult or discriminating than items with three distractors. Preparation of examination questions targeting higher cognitive levels will increase the likelihood of constructing discriminating items. Use of implausible distractors to complete a five-option multiple choice question does not strengthen the discrimination value.Keywords
Funding Information
- Kansas State University
This publication has 37 references indexed in Scilit:
- Multiple choice questions are superior to extended matching questions to identify medicine and biomedical sciences students who perform poorlyPerspectives on Medical Education, 2013
- Incorporation of Bloom’s Taxonomy into Multiple-Choice Examination Questions for a Pharmacotherapeutics CourseAmerican Journal of Pharmaceutical Education, 2012
- Rarely selected distractors in high stakes medical multiple-choice examinations and their recognition by item authors: a simulation and surveyBMC Medical Education, 2010
- Job requirements compared to medical school education: differences between graduates from problem-based learning and conventional curriculaBMC Medical Education, 2010
- A primer on classical test theory and item response theory for assessments in medical educationMedical Education, 2009
- An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysisBMC Medical Education, 2009
- Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paperBMC Medical Education, 2007
- Construct-irrelevant Variance and Flawed Test QuestionsAcademic Medicine, 2002
- Quality Multiple-Choice Test Questions: Item-Writing Guidelines and an Analysis of Auditing TestbanksJournal of Education for Business, 1997
- Item Type and Cognitive Ability Measured: The Validity Evidence for Multiple True-False Items in Medical Specialty CertificationApplied Measurement in Education, 1995