Administrators' Uses of Teacher Observation Protocol in Different Rating Contexts
Open Access
- 27 April 2018
- journal article
- research article
- Published by Wiley in ETS Research Report Series
- Vol. 2018 (1) , 1-19
Abstract
Teacher observations are being used for high‐stakes purposes in states across the country, and administrators often serve as raters in teacher evaluation systems. This paper examines how the cognitive aspects of administrators' use of an observation instrument, a modified version of Charlotte Danielson's Framework for Teaching, interact with the complex and dynamic rating contexts in applied settings. Findings suggest that administrators' rating strategies and rating approaches vary as the characteristics of the rating contexts differ. Even shortly after training (and more so as time passed), raters used reasoning strategies not supported by their training to make scoring decisions. We discuss the implications of the findings for the training of raters and the development of evaluation systems in high‐stakes contexts.Keywords
This publication has 31 references indexed in Scilit:
- Effect of Observation Mode on Measures of Secondary Mathematics TeachingEducational and Psychological Measurement, 2013
- When Rater Reliability Is Not EnoughEducational Researcher, 2012
- Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category UseJournal of Educational Measurement, 2009
- The Sensitivity of Value‐Added Teacher Effect Estimates to Different Mathematics Achievement MeasuresJournal of Educational Measurement, 2007
- Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State ExaminationJournal of Educational Measurement, 2001
- Portrait of the “Ideal Principal”: Context and SelfNASSP Bulletin, 2000
- The Stability of Rater Severity in Large‐Scale Assessment ProgramsJournal of Educational Measurement, 2000
- Understanding Scoring Reliability: Experiments in Calibrating Essay ReadersJournal of Educational Statistics, 1988
- Teacher Evaluation in the Organizational Context: A Review of the LiteratureReview of Educational Research, 1983
- Generalizability of Measures of Teaching BehaviorReview of Educational Research, 1976