Big Data Pre-Processing: Closing the Data Quality Enforcement Loop
Authors:
- Ikbal TalebITIkbal Taleb
- Mohamed Adel SerhaniMSMohamed Adel Serhani
- 1 June 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 498-501
Abstract
In the Big Data Era, data is the core for any governmental, institutional, and private organization. Efforts were geared towards extracting highly valuable insights that cannot happen if data is of poor quality. Therefore, data quality (DQ) is considered as a key element in Big data processing phase. In this stage, low quality data is not penetrated to the Big Data value chain. This paper, addresses the data quality rules discovery (DQR) after the evaluation of quality and prior to Big Data pre-processing. We propose a DQR discovery model to enhance and accurately target the pre-processing activities based on quality requirements. We defined, a set of pre-processing activities associated with data quality dimensions (DQD's) to automatize the DQR generation process. Rules optimization are applied on validated rules to avoid multi-passes pre-processing activities and eliminates duplicate rules. Conducted experiments showed an increased quality scores after applying the discovered and optimized DQR's on data.Keywords
This publication has 9 references indexed in Scilit:
- Quality Evaluation for Big Data: A Scalable Assessment Approach and First Evaluation Results2016
- Big Data Quality: A Quality Dimensions Evaluation2016
- Process-Driven Data Quality Management -- An Application of the Combined Conceptual Life Cycle Model2014
- A Data Quality in Use Model for Big DataLecture Notes in Computer Science, 2014
- Data quality: A survey of data quality dimensions2012
- An Efficient and Robust Approach for Discovering Data Quality Rules2010
- Discovering data quality rulesProceedings of the VLDB Endowment, 2008
- Dependencies revisited for improving data quality2008
- Crafting Rules: Context-Reflective Data Quality Problem SolvingJournal of Management Information Systems, 2003