RCTs to Scale: Comprehensive Evidence From Two Nudge Units
Authors:
Top Cited Papers
- 1 January 2022
- journal article
- research article
- Published by The Econometric Society in Econometrica
- Vol. 90 (1) , 81-116
Abstract
Nudge interventions have quickly expanded from academic studies to larger implementation in so‐called Nudge Units in governments. This provides an opportunity to compare interventions in research studies, versus at scale. We assemble a unique data set of 126 RCTs covering 23 million individuals, including all trials run by two of the largest Nudge Units in the United States. We compare these trials to a sample of nudge trials in academic journals from two recent meta‐analyses. In the Academic Journals papers, the average impact of a nudge is very large—an 8.7 percentage point take‐up effect, which is a 33.4% increase over the average control. In the Nudge Units sample, the average impact is still sizable and highly statistically significant, but smaller at 1.4 percentage points, an 8.0% increase. We document three dimensions which can account for the difference between these two estimates: (i) statistical power of the trials; (ii) characteristics of the interventions, such as topic area and behavioral channel; and (iii) selective publication. A meta‐analysis model incorporating these dimensions indicates that selective publication in the Academic Journals sample, exacerbated by low statistical power, explains about 70 percent of the difference in effect sizes between the two samples. Different nudge characteristics account for most of the residual difference.Keywords
This publication has 22 references indexed in Scilit:
- What Motivates Effort? Evidence and Expert ForecastsReview of Economic Studies, 2017
- The behavioralist as tax collector: Using natural field experiments to enhance tax complianceJournal of Public Economics, 2017
- Evaluating replicability of laboratory experiments in economicsScience, 2016
- Psychological Frictions and the Incomplete Take-Up of Social Benefits: Evidence from an IRS Field ExperimentAmerican Economic Review, 2015
- Site Selection Bias in Program Evaluation *The Quarterly Journal of Economics, 2015
- Publication bias in the social sciences: Unlocking the file drawerScience, 2014
- P-curve: A key to the file-drawer.Journal of Experimental Psychology: General, 2014
- The Experimental Approach to Development EconomicsAnnual Review of Economics, 2009
- Advanced methods in meta‐analysis: multivariate approach and meta‐regressionStatistics in Medicine, 2002
- Recent Developments in Computer-Assisted Analysis of Mixtures1998