An Hybrid Approach to Quality Evaluation across Big Data Value Chain
Authors:
- Mohamed Adel SerhaniMSMohamed Adel Serhani
- Hadeel T. El KassabiHKHadeel T. El Kassabi
- Ikbal TalebITIkbal Taleb
- Alramzana NujumANAlramzana Nujum
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 418-425
Abstract
While the potential benefits of Big Data adoption are significant, and some initial successes have already been realized, there remain many research and technical challenges that must be addressed to fully realize this potential. The Big Data processing, storage and analytics, of course, are major challenges that are most easily recognized. However, there are additional challenges related for instance to Big Data collection, integration, and quality enforcement. This paper proposes a hybrid approach to Big Data quality evaluation across the Big Data value chain. It consists of assessing first the quality of Big Data itself, which involve processes such as cleansing, filtering and approximation. Then, assessing the quality of process handling this Big Data, which involve for example processing and analytics process. We conduct a set of experiments to evaluate Quality of Data prior and after its pre-processing, and the Quality of the pre-processing and processing on a large dataset. Quality metrics have been measured to access three Big Data quality dimensions: accuracy, completeness, and consistency. The results proved that combination of data-driven and process-driven quality evaluation lead to improved quality enforcement across the Big Data value chain. Hence, we recorded high prediction accuracy and low processing time after we evaluate 6 well-known classification algorithms as part of processing and analytics phase of Big Data value chain.Keywords
This publication has 24 references indexed in Scilit:
- Overview of data quality challenges in the context of Big Data2015
- Data quality issues in big data2015
- The Challenges of Data Quality and Data Quality Assessment in the Big Data EraData Science Journal, 2015
- Big Data Cleaning2014
- NADEEFProceedings of the VLDB Endowment, 2013
- Towards a Quality-centric Big Data Architecture for Federated Sensor Services2013
- Maximizing the Value of Ecological Data with Structured Metadata: An Introduction to Ecological Metadata Language (EML) and Principles for Metadata CreationThe Bulletin of the Ecological Society of America, 2005
- A model for web services discovery with QoSACM SIGecom Exchanges, 2003
- Data quality assessmentCommunications of the ACM, 2002
- Metadata standards for data warehousingACM SIGMOD Record, 2000