How Credible Is the Evidence, and Does It Matter? An Analysis of the Program Assessment Rating Tool
- 11 November 2011
- journal article
- research article
- Published by Wiley in Public Administration Review
- Vol. 72 (1) , 123-134
Abstract
This research empirically assesses the quality of evidence that agencies provided to the Office of Management and Budget in the application of the Program Assessment Rating Tool (PART), introduced in 2002 to more rigorously, systematically, and transparently assess public program effectiveness and hold agencies accountable for results by tying them to the executive budget formulation process and program funding. Evidence submitted by 95 programs administered by the U.S. Department of Health and Human Services for the PART assessment is analyzed using measures that capture the quality of evidence and methods used by programs and information on characteristics of agencies that might relate to program results and government funding decisions. The study finds that of those programs offering some evidence, most was internal and qualitative, and about half did not assess how their performance compared to other government or private programs with similar objectives. Programs were least likely to provide externally generated evidence of their performance relative to long‐term and annual performance goals. Importantly, overall PART and results scores were (statistically) significantly lower for programs that failed to provide quantitative evidence and did not use long‐term measures, baseline measures or targets, or independent evaluations. Although the PART program results ratings and overall PART scores had no discernible consequences for program funding over time, the PART assessments appeared to take seriously the evaluation of evidence quality, a positive step forward in recent efforts to base policy decisions on more rigorous evidence.Keywords
This publication has 9 references indexed in Scilit:
- OF POLITICS AND PURPOSE: POLITICAL SALIENCE AND GOAL AMBIGUITY OF US FEDERAL AGENCIESPublic Administration, 2009
- Evidence-Based Policy and Performance ManagementThe American Review of Public Administration, 2007
- Three Bush Administration Management Reform Initiatives: The President’s Management Agenda, Freedom to Manage Legislative Proposals, and the Program Assessment Rating ToolPublic Administration Review, 2007
- Does Performance Budgeting Work? An Examination of the Office of Management and Budget’s PART ScoresPublic Administration Review, 2006
- Why PART? The Institutional Politics of Presidential Budget ReformJournal of Public Administration Research and Theory, 2006
- Goal Ambiguity and Organizational Performance in U.S. Federal AgenciesJournal of Public Administration Research and Theory, 2005
- Is it ‘what works’ that matters? Evaluation and evidence‐based policy‐makingResearch Papers in Education, 2003
- The Government Performance and Results Act and the Tradition of Federal Management Reform: Square Pegs in Round Holes?Journal of Public Administration Research and Theory, 2000
- The New Public Management: How to Transform a Theme into a LegacyPublic Administration Review, 1998