Do Estimated Impacts on Earnings Depend on the Source of the Data Used to Measure Them? Evidence From Previous Social Experiments
Authors:
- 31 December 2014
- journal article
- research article
- Published by SAGE Publications in Evaluation Review
- Vol. 39 (2) , 179-228
Abstract
Background: Impact evaluations draw their data from two sources, namely, surveys conducted for the evaluation or administrative data collected for other purposes. Both types of data have been used in impact evaluations of social programs. Objective: This study analyzes the causes of differences in impact estimates when survey data and administrative data are used to evaluate earnings impacts in social experiments and discusses the differences observed in eight evaluations of social experiments that used both survey and administrative data. Results: There are important trade-offs between the two data sources. Administrative data are less expensive but may not cover all income and may not cover the time period desired, while surveys can be designed to avoid these problems. We note that errors can be due to nonresponse or reporting, and errors can be balanced between the treatment and the control groups or unbalanced. We find that earnings are usually higher in survey data than in administrative data due to differences in coverage and likely overreporting of overtime hours and pay in survey data. Evaluations using survey data usually find greater impacts, sometimes much greater. Conclusions: The much lower cost of administrative data make their use attractive, but they are still subject to underreporting and other problems. We recommend further evaluations using both types of data with investigative audits to better understand the sources and magnitudes of errors in both survey and administrative data so that appropriate corrections to the data can be made.Keywords
This publication has 17 references indexed in Scilit:
- The implications of differences between employer and worker employment/earnings reports for policy evaluationJournal of Policy Analysis and Management, 2007
- Measurement Error and Misclassification: A Comparison of Survey and Administrative DataJournal of Labor Economics, 2007
- Measuring Program Impacts on Earnings and Employment: Do Unemployment Insurance Wage Reports from Employers Agree with Surveys of Individuals?Journal of Labor Economics, 1999
- Employer Tax Evasion in the Unemployment Insurance ProgramJournal of Labor Economics, 1996
- Errors in Survey Reports of Earnings, Hours Worked, and Hourly WagesJournal of the American Statistical Association, 1993
- The Extent of Measurement Error in Longitudinal Earnings Data: Do Two Wrongs Make a Right?Journal of Labor Economics, 1991
- An Investigation of the Extent and Consequences of Measurement Error in Labor-Economic Survey DataJournal of Labor Economics, 1985
- Accuracy of Response in Labor Market Surveys: Evidence and ImplicationsJournal of Labor Economics, 1983
- Systematic Misreporting and Effects of Income Maintenance Experiments on Work Effort: Evidence from the Seattle-Denver ExperimentJournal of Labor Economics, 1983
- Underreporting and Experimental Effects on Work Effort: Evidence from the Gary Income Maintenance ExperimentReview of Economics and Statistics, 1981