SCODED: Statistical Constraint Oriented Data Error Detection
Authors:
Open Access
- 31 May 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 845-860
Abstract
Statistical Constraints (SCs) play an important role in statistical modeling and analysis. This paper brings the concept to data cleaning and studies how to leverage SCs for error detection. SCs provide a novel approach that has various application scenarios and works harmoniously with downstream statistical modeling. Entailment relationships between SCs and integrity constraints provide analytical insight into SCs. We develop SCODED, an SC-Oriented Data Error Detection system, comprising two key components: (1) SC Violation Detection : checks whether an SC is violated on a given dataset, and (2) Error Drill Down : identifies the top-k records that contribute most to the violation of an SC. Experiments on synthetic and real-world data show that SCs are effective in detecting data errors that violate them, compared to state-of-the-art approaches.Keywords
This publication has 43 references indexed in Scilit:
- On the conditional independence implication problem: A lattice-theoretic approachArtificial Intelligence, 2013
- A comparative analysis of Spearman's rho and Kendall's tau in normal and contaminated normal modelsSignal Processing, 2013
- A Regression-Based Adjusted Plus-Minus Statistic for NHL PlayersJournal of Quantitative Analysis in Sports, 2011
- Medical data mining: insights from winning two competitionsData Mining and Knowledge Discovery, 2009
- On the relationship between Spearman's rho and Kendall's tau for pairs of continuous random variablesJournal of Statistical Planning and Inference, 2007
- TinyDB: an acquisitional query processing system for sensor networksACM Transactions on Database Systems, 2005
- Logical and Algorithmic Properties of Conditional Independence and Graphical ModelsThe Annals of Statistics, 1993
- Hedonic housing prices and the demand for clean airJournal of Environmental Economics and Management, 1978
- Multivalued dependencies and a new normal form for relational databasesACM Transactions on Database Systems, 1977
- A Computer Method for Calculating Kendall's Tau with Ungrouped DataJournal of the American Statistical Association, 1966