A Framework for Addressing Instrumentation Biases When Using Observation Systems as Outcome Measures in Instructional Interventions
Open Access
- 17 June 2022
- journal article
- research article
- Published by Taylor & Francis in Journal of Research on Educational Effectiveness
- Vol. 16 (1) , 162-188
Abstract
Many educational interventions seek to change teachers’ instructional practice. Standards-based observation systems are a common and useful tool to understand these interventions’ impact, but the process of measuring instructional change with observation systems is highly complex. This paper introduces a framework for examining and understanding potential instrumentation biases that arise when evaluations use observation systems to understand instructional change. The framework systematizes two processes that all studies must undertake: (1) the process of operationalizing the construct of teaching quality, and (2) the process of data collection. A study that engages in these processes generates observation scores that capture their own raters’ perspectives on specific segments of instruction. These scores must be generalized to draw conclusions about the intended constructs and settings. Systematizing these two processes highlights the necessary steps of a validity argument supporting evaluation conclusions and the instrumentation biases that threaten such conclusions. The framework is illustrated with an example from our recent work, which sought to understand instructional change since the adoption of the Common Core State Standards (CCSS).Keywords
Funding Information
- Spencer Foundation
- W.T. Grant foundation for a project entitled “Under Construction: The Rise, Spread, and Consequences of the Common Core State Standards Initiative in American Education.”
This publication has 51 references indexed in Scilit:
- Effect of Observation Mode on Measures of Secondary Mathematics TeachingEducational and Psychological Measurement, 2013
- Observer ratings of instructional quality: Do they fulfill what they promise?Learning and Instruction, 2012
- Rater Cognition: Implications for ValidityEducational Measurement: Issues and Practice, 2012
- An Argument Approach to Observation Protocol ValidityEducational Assessment, 2012
- False-Positive PsychologyPsychological Science, 2011
- Measures Inherent to Treatments in Program Effectiveness ReviewsJournal of Research on Educational Effectiveness, 2011
- Within-Day Variability in the Quality of Classroom Interactions during Third and Fifth GradeThe Elementary School Journal, 2011
- Embracing the Complexity of Instruction: A Study of the Effects of Teachers' Instruction on Students' Reading ComprehensionScientific Studies of Reading, 2011
- Rater training for performance appraisal: A quantitative reviewJournal of Occupational and Organizational Psychology, 1994
- On Simpson's Paradox and the Sure-Thing PrincipleJournal of the American Statistical Association, 1972