Differential Data Quality Verification on Partitioned Data
Authors:
- 1 April 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1940-1945
Abstract
Modern companies and institutions rely on data to guide every single decision. Missing or incorrect information seriously compromises any decision process. In previous work, we presented Deequ, a Spark-based library for automating the verification of data quality at scale. Deequ provides a declarative API, which combines common quality constraints with user-defined validation code, and thereby enables "unit tests for data". However, we found that the previous computational model of Deequ is not flexible enough for many scenarios in modern data pipelines, which handle large, partitioned datasets. Such scenarios require the evaluation of dataset-level quality constraints after individual partition updates, without having to re-read already processed partitions. Additionally, such scenarios often require the verification of data quality on select combinations of partitions. We therefore present a differential generalization of the computational model of Deequ, based on algebraic states with monoid properties. We detail how to efficiently implement the corresponding operators and aggregation functions in Apache Spark. Furthermore, we show how to optimize the resulting workloads to minimize the required number of passes over the data, and empirically validate that our approach decreases the runtimes for updating data metrics under data changes and for different combinations of partitions.Keywords
This publication has 10 references indexed in Scilit:
- Automating large-scale data quality verificationProceedings of the VLDB Endowment, 2018
- TFX2017
- Probabilistic demand forecasting at scaleProceedings of the VLDB Endowment, 2017
- Data Cleaning2016
- SummingbirdProceedings of the VLDB Endowment, 2014
- Discovering denial constraintsProceedings of the VLDB Endowment, 2013
- HyperLogLog in practice2013
- Space-efficient online computation of quantile summariesACM SIGMOD Record, 2001
- Efficient view maintenance at data warehouses1997
- Algorithms for Computing the Sample Variance: Analysis and RecommendationsThe American Statistician, 1983