Data profiling
Authors:
- 1 May 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1432-1435
Abstract
One of the crucial requirements before consuming datasets for any application is to understand the dataset at hand and its metadata. The process of metadata discovery is known as data profiling. Profiling activities range from ad-hoc approaches, such as eye-balling random subsets of the data or formulating aggregation queries, to systematic inference of structural information and statistics of a dataset using dedicated profiling tools. In this tutorial, we highlight the importance of data profiling as part of any data-related use-case, and discuss the area of data profiling by classifying data profiling tasks and reviewing the state-of-the-art data profiling systems and techniques. In particular, we discuss hard problems in data profiling, such as algorithms for dependency discovery and profiling algorithms for dynamic data and streams. We conclude with directions for future research in the area of data profiling. This tutorial is based on our survey on profiling relational data [1].Keywords
This publication has 21 references indexed in Scilit:
- Temporal rules discovery for web data cleaningProceedings of the VLDB Endowment, 2015
- Data profiling with metanomeProceedings of the VLDB Endowment, 2015
- Divide & conquer-based inclusion dependency discoveryProceedings of the VLDB Endowment, 2015
- DFD2014
- Detecting unique column combinations on dynamic data2014
- Profiling and mining RDF data with ProLOD++2014
- Scalable discovery of unique column combinationsProceedings of the VLDB Endowment, 2013
- Advancing the discovery of unique column combinations2011
- Efficiently Detecting Inclusion Dependencies2007
- Tane: An Efficient Algorithm for Discovering Functional and Approximate DependenciesThe Computer Journal, 1999