Evaluating Modes of Observations Using Hierarchical Signal Detection Rater Models
- 11 December 2022
- journal article
- research article
- Published by Taylor & Francis in Multivariate Behavioral Research
- Vol. 58 (4) , 843-858
Abstract
Scoring of performance can be based on different modes of observation, which can include live and videotaped observations. Although live observations have been the traditional format of measuring examinee performance in education and in healthcare, videotaped observations provide educators and testing agencies the promise of unbiased and standardized evaluations, offering practical solutions to limitations of real-time scoring. This study proposes a measurement model taking into account different modes of observation, using an extension of the hierarchical rater model based on signal detection theory (HRM-SDT). A hierarchical rater model is motivated by the nested structure of scores assigned by raters - scores assigned by raters become indicators of performance quality, which in turn become indicators of examinee ability. This study extends the hierarchical structure of the scoring process to include modes of observation, which serves as an intermediary level between the latent categorical indicator of performance quality and examinee ability, forming a three-level HRM-SDT. Analyses based on real-world data showed differences in the quality of scores from live observations and videotaped recordings. Compared to the traditional HRM-SDT, the overall model fit improved when including modes of observation. Simulations using different sample sizes and conditions provide implications for uses of this model.Keywords
This publication has 27 references indexed in Scilit:
- Effect of Observation Mode on Measures of Secondary Mathematics TeachingEducational and Psychological Measurement, 2013
- Evaluating Intraoperative Laparoscopic Skill: Direct Observation Versus Blinded Videotaped PerformancesSurgical Innovation, 2007
- Assessing the technical skills of surgical traineesBritish Journal of Surgery, 2005
- The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment DataJournal of Educational and Behavioral Statistics, 2002
- A Cautionary Note on IRT‐Based Linking of Tests With Polytomous ItemsJournal of Educational Measurement, 1999
- Assessment of clinical skills with standardized patients: State of the artTeaching and Learning in Medicine, 1990
- Affect and appraisal accuracy: Liking as an integral dimension in evaluating performance.Journal of Applied Psychology, 1986
- THE CONTROL OF BIAS IN RATINGS: A THEORY OF RATINGPersonnel Psychology, 1982
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960