Do Typical RCTs of Education Interventions Have Sufficient Statistical Power for Linking Impacts on Teacher Practice and Student Achievement Outcomes?
Author:
- 1 August 2011
- journal article
- Published by American Educational Research Association (AERA) in Journal of Educational and Behavioral Statistics
- Vol. 36 (4) , 441-471
Abstract
For RCTs of education interventions, it is often of interest to estimate associations between student and mediating teacher practice outcomes, to examine the extent to which the study’s conceptual model is supported by the data, and to identify specific mediators that are most associated with student learning. This article develops statistical power formulas for such exploratory analyses under clustered school-based RCTs using ordinary least squares (OLS) and instrumental variable (IV) estimators and uses these formulas to conduct a simulated power analysis. The power analysis finds that for currently available mediators, the OLS approach will yield precise estimates of associations between teacher practice measures and student test score gains only if the sample contains about 150 to 200 study schools. The IV approach, which can adjust for potential omitted variable and simultaneity biases, has very little statistical power for mediator analyses. For typical RCT evaluations, these results may have design implications for the scope of the data collection effort for obtaining costly teacher practice mediators.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Intraclass Correlation Values for Planning Group-Randomized Trials in EducationEducational Evaluation and Policy Analysis, 2007
- Experimental Analysis of Neighborhood EffectsEconometrica, 2007
- Teachers and Student Achievement in the Chicago Public High SchoolsJournal of Labor Economics, 2007
- Principals as Agents: Subjective Performance Measurement in Education2005
- Models for Value-Added Modeling of Teacher EffectsJournal of Educational and Behavioral Statistics, 2004
- Power analysis and determination of sample size for covariance structure modeling.Psychological Methods, 1996
- Identification of Causal Effects Using Instrumental VariablesJournal of the American Statistical Association, 1996
- Identification and Estimation of Local Average Treatment EffectsEconometrica, 1994
- Causal Inference, Path Analysis, and Recursive Structural Equations ModelsSociological Methodology, 1988
- The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Journal of Personality and Social Psychology, 1986