Outliers in official statistics
Author:
Open Access
- 24 October 2020
- journal article
- research article
- Published by Springer Nature in Japanese Journal of Statistics and Data Science
- Vol. 3 (2) , 669-691
Abstract
The purpose of this manuscript is to provide a survey on the important methods addressing outliers while producing official statistics. Outliers are often unavoidable in survey statistics. They may reduce the information of survey datasets and distort estimation on each step of the survey statistics production process. This paper defines outliers to be focused on each production step and introduces practical methods to cope with them. The statistical production process is roughly divided into the following three steps. The first step is data cleaning, and outliers to be focused are that may contain mistakes to be corrected. Robust estimators of a mean vector and covariance matrix are introduced for the purpose. The next step is imputation. Among a variety of imputation methods, regression and ratio imputation are the subjects in this paper. Outliers to be focused on in this step are not erroneous but have extreme values that may distort parameter estimation. Robust estimators that are not affected by remaining outliers are introduced. The final step is estimation and formatting. We have to be careful about outliers that have extreme values with large design weights since they have a considerable influence on the final statistics products. Weight calibration methods controlling the influence are discussed based on the robust weights obtained in the previous imputation step. A few examples of practical application are also provided briefly, although multivariate outlier detection methods introduced in this paper are mostly in the research stage in the field of official statistics.Keywords
Funding Information
- Japan Society for the Promotion of Science (JP16H2013)
This publication has 39 references indexed in Scilit:
- Multivariate Outlier Detection in Incomplete Survey Data: The Epidemic Algorithm and Transformed Rank CorrelationsJournal of the Royal Statistical Society Series A: Statistics in Society, 2004
- Nearest-Neighbor Variance Estimation (NNVE)Journal of the American Statistical Association, 2002
- A Fast Algorithm for the Minimum Covariance Determinant EstimatorTechnometrics, 1999
- Introduction to Robust Estimation and Hypothesis TestingJournal of the American Statistical Association, 1998
- High Breakdown-Point and High Efficiency Robust Estimates for RegressionThe Annals of Statistics, 1987
- Outlier Robust Finite Population EstimationJournal of the American Statistical Association, 1986
- Least Median of Squares RegressionJournal of the American Statistical Association, 1984
- Minimax Aspects of Bounded-Influence RegressionJournal of the American Statistical Association, 1983
- The Fitting of Power Series, Meaning Polynomials, Illustrated on Band-Spectroscopic DataTechnometrics, 1974
- A Generalization of Sampling Without Replacement from a Finite UniverseJournal of the American Statistical Association, 1952