From noise to knowledge: Improving evidentiary standards for program efficacy to better inform public policy and management decisions
Authors:
Open Access
- 23 July 2023
- journal article
- research article
- Published by Wiley in Public Administration Review
- Vol. 83 (5) , 1051-1071
Abstract
Current approaches employed by U.S.‐based clearinghouses to rate the efficacy of interventions to address social problems typically do not result in sufficient information to help practitioners. Current standards of evidence employed across the United States apply a positivist notion of validity with quantitative research criteria that discourage answering important how and why questions, explicitly privilege quantitative/RCT evidence, offer few contextual insights, and rarely discuss disparities in outcomes across participants differing by race, gender, and ethnicity. We offer a set of standards of evidence to assess qualitative and mixed methods studies, as well as RCTs and quasi‐experimental designs, and probe the extent to which the studies address context and equity. We applied our proposed new standards of evidence to all intervention studies rated as the highest quality by the What Works Clearinghouse (WWC) sponsored by the U.S. Department of Education from 2017 to 2021 to demonstrate the usefulness of our standards.Keywords
This publication has 47 references indexed in Scilit:
- A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies ReviewsInternational Journal of Nursing Studies, 2009
- What Are Effect Sizes and Why Do We Need Them?Child Development Perspectives, 2008
- Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groupsInternational Journal for Quality in Health Care, 2007
- Studying the Complicated Matter of What Works: Evidence-Based Research and the Problem of PracticeAdult Education Quarterly, 2006
- Mixed Methods Research: A Research Paradigm Whose Time Has ComeEducational Researcher, 2004
- Criteria for evaluating evidence on public health interventionsJournal of Epidemiology and Community Health, 2002
- Nobody knows my name: In praise of african american evaluators who were responsiveNew Directions for Evaluation, 2001
- Qualitative research in health care: Assessing quality in qualitative researchBMJ, 2000
- The Social Construction of ValidityQualitative Inquiry, 1995
- Factors relevant to the validity of experiments in social settings.Psychological Bulletin, 1957