Evaluating Organizational Effectiveness: A New Strategy to Leverage Multisite Randomized Trials for Valid Assessment
- 25 April 2026
- journal article
- Published by SAGE Publications in American Journal of Evaluation
- Vol. 47 (3) , 392-420
Abstract
Determining which organizations are more effective in implementing an intervention program is essential for theoretically and empirically characterizing exemplary practice and for intervening to enhance the capacity of ineffective ones. However, sites differ in their local ecological conditions including client composition, availability of alternative programs, and broader community context. Applying the causal inference framework, this study proposes a formal mathematical definition of an organization's local relative effectiveness attributable solely to malleable organizational practice. Capitalizing on multisite randomized trials, the identification leverages observed control group outcomes that capture some of the confounding impacts of otherwise unmeasured contextual variation. We propose a two-step mixed-effects modeling procedure that adjusts for pre-existing between-site variation. Monte Carlo simulations demonstrate its superior performance compared to conventional methods. We apply the new strategy to an evaluation of Job Corps centers nationwide serving disadvantaged youths.Keywords
All Related Versions
Funding Information
- Spencer Foundation (202400131)
- Institute of Education Sciences (R305D200031)
This publication has 78 references indexed in Scilit:
- UNPACKING COMPLEX MEDIATION MECHANISMS AND THEIR HETEROGENEITY BETWEEN SITES IN A JOB CORPS EVALUATIONJournal of Policy Analysis and Management, 2020
- Long‐Run Labor Market Effects of the Job Corps Program: Evidence from a Nationally Representative ExperimentJournal of Policy Analysis and Management, 2020
- Extending inferences from a randomized trial to a new target populationStatistics in Medicine, 2020
- A flexible, interpretable framework for assessing sensitivity to unmeasured confoundingStatistics in Medicine, 2016
- Making Decisions with Imprecise Performance Measures2015
- Causality in a Social World2015
- A CONCEPTUAL FRAMEWORK FOR STUDYING THE SOURCES OF VARIATION IN PROGRAM EFFECTSJournal of Policy Analysis and Management, 2014
- When Do Regression-Adjusted Performance Measures Track Longer-Term Program Impacts? A Case Study for Job CorpsJournal of Policy Analysis and Management, 2014
- Do Job Corps performance measures track program impacts?Journal of Policy Analysis and Management, 2008
- Linking program implementation and effectiveness: Lessons from a pooled sample of welfare‐to‐work experimentsJournal of Policy Analysis and Management, 2003