We need more replication research – A case for test-retest reliability
Authors:
Open Access
- 7 April 2017
- journal article
- review article
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 6 (3) , 158-164
Abstract
Following debates in psychology on the importance of replication research, we have also started to see pleas for a more prominent role for replication research in medical education. To enable replication research, it is of paramount importance to carefully study the reliability of the instruments we use. Cronbach’s alpha has been the most widely used estimator of reliability in the field of medical education, notably as some kind of quality label of test or questionnaire scores based on multiple items or of the reliability of assessment across exam stations. However, as this narrative review outlines, Cronbach’s alpha or alternative reliability statistics may complement but not replace psychometric methods such as factor analysis. Moreover, multiple-item measurements should be preferred above single-item measurements, and when using single-item measurements, coefficients as Cronbach’s alpha should not be interpreted as indicators of the reliability of a single item when that item is administered after fundamentally different activities, such as learning tasks that differ in content. Finally, if we want to follow up on recent pleas for more replication research, we have to start studying the test-retest reliability of the instruments we use.Keywords
This publication has 38 references indexed in Scilit:
- Making sense of Cronbach's alphaInternational Journal of Medical Education, 2011
- A Primer on the Validity of Assessment InstrumentsJournal of Graduate Medical Education, 2011
- On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s AlphaPsychometrika, 2009
- What is being assessed and why it matters: The impact of transient error on trait research.Journal of Personality and Social Psychology, 2009
- Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much BetterPerspectives on Psychological Science, 2008
- Congeneric and (Essentially) Tau-Equivalent Estimates of Score ReliabilityEducational and Psychological Measurement, 2006
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of ReliabilityEducational and Psychological Measurement, 1973
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- Coefficient Alpha and the Internal Structure of TestsPsychometrika, 1951