Flaws in Evaluations of Social Programs
Authors:
- 21 August 2014
- journal article
- Published by SAGE Publications in Evaluation Review
- Vol. 38 (5) , 359-387
Abstract
Background: This article describes eight flaws that occur in impact evaluations. Method: The eight flaws are grouped into four categories on how they affect impact estimates: statistical imprecision; biases; failure of impact estimates to measure effects of the planned treatment; and flaws that result from weakening an evaluation design. Each flaw is illustrated with examples from social experiments. Although these illustrations are from randomized controlled trials (RCTs), they can occur in any type of evaluation; we use RCTs to illustrate because people sometimes assume that RCTs might be immune to such problems. A summary table lists the flaws, indicates circumstances under which they occur, notes their potential seriousness, and suggests approaches for minimizing them. Results: Some of the flaws result in minor hurdles, while others cause evaluations to fail—that is, the evaluation is unable to provide a valid test of the hypothesis of interest. The flaws that appear to occur most frequently are response bias resulting from attrition, failure to adequately implement the treatment as designed, and too small a sample to detect impacts. The third of these can result from insufficient marketing, too small an initial target group, disinterest on the part of the target group in participating (if the treatment is voluntary), or attrition. Conclusion: To a considerable degree, the flaws we discuss can be minimized. For instance, implementation failures and too small a sample can usually be avoided with sufficient planning, and response bias can often be mitigated—for example, through increased follow-up efforts in conducting surveys.Keywords
This publication has 24 references indexed in Scilit:
- Replication issues in social experiments: lessons from US labor market programsJournal for Labour Market Research, 2013
- Is Decentralized Iron Fortification a Feasible Option to Fight Anemia among the Poorest?2011
- In Pursuit of Balance: Randomization in Practice in Development Field ExperimentsAmerican Economic Journal: Applied Economics, 2009
- Toward a more public discussion of the ethics of federal social program evaluationJournal of Policy Analysis and Management, 2005
- Linking program implementation and effectiveness: Lessons from a pooled sample of welfare‐to‐work experimentsJournal of Policy Analysis and Management, 2003
- The Effect of High School Matriculation Awards: Evidence from Randomized Trials2002
- Minimum Detectable EffectsEvaluation Review, 1995
- Avoiding Type III Errors in Health Education Program Evaluations: A Case StudyHealth Education Quarterly, 1985
- Comparative evaluation of the Job Club program with welfare recipientsJournal of Vocational Behavior, 1980
- Job-finding club: A group-assisted program for obtaining employmentBehaviour Research and Therapy, 1975