Rapid Identification of Column Heterogeneity
- 1 December 2006
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE International Conference on Data Mining (ICDM)
- No. 15504786,p. 159-170
Abstract
Data quality is a serious concern in every data management application, and a variety of quality measures have been proposed, e.g., accuracy, freshness and completeness, to capture common sources of data quality degradation. We identify and focus attention on a novel measure, column heterogeneity, that seeks to quantify the data quality problems that can arise when merging data from different sources. We identify desiderata that a column heterogeneity measure should intuitively satisfy, and describe our technique to quantify database column heterogeneity based on using a novel combination of cluster entropy and soft clustering. Finally, we present detailed experimental results, using diverse data sets of different types, to demonstrate that our approach provides a robust mechanism for identifying and quantifying database column heterogeneity.Keywords
This publication has 8 references indexed in Scilit:
- Flexible String Matching Against Large Databases in Practice2004
- Data quality and data cleaning2003
- Exploratory Data Mining and Data Cleaning2003
- Mining database structure; or, how to build a data quality browser2002
- Elements of Information Theory2001
- Estimating the Dimension of a ModelThe Annals of Statistics, 1978
- Maximum Likelihood from Incomplete Data Via the EM AlgorithmJournal of the Royal Statistical Society Series B: Statistical Methodology, 1977
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974