Mostly harmless simulations? Using Monte Carlo studies for estimator selection
Open Access
- 25 June 2019
- journal article
- research article
- Published by Wiley in Journal of Applied Econometrics
- Vol. 34 (6) , 893-910
Abstract
Summary: We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real‐world setting where estimator performance is known. Both approaches are worse than random at selecting estimators that minimize absolute bias. They are better when selecting estimators that minimize mean squared error. However, using a simple bootstrap is at least as good and often better. For now, researchers would be best advised to use a range of estimators and compare estimates for robustness.Keywords
This publication has 43 references indexed in Scilit:
- Bias-Corrected Matching Estimators for Average Treatment EffectsJournal of Business & Economic Statistics, 2011
- The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studiesStatistics in Medicine, 2010
- On the Specification of Propensity Scores, With Applications to the Analysis of Trade PoliciesJournal of Business & Economic Statistics, 2009
- Generalized least squares inference in panel and multilevel models with serial correlation and fixed effectsJournal of Econometrics, 2007
- Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo EvidenceReview of Economics and Statistics, 2004
- How Much Should We Trust Differences-In-Differences Estimates?The Quarterly Journal of Economics, 2004
- Efficient Estimation of Average Treatment Effects Using the Estimated Propensity ScoreEconometrica, 2003
- Propensity Score-Matching Methods for Nonexperimental Causal StudiesReview of Economics and Statistics, 2002
- Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training ProgramsJournal of the American Statistical Association, 1999
- Choosing among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower TrainingJournal of the American Statistical Association, 1989