Unsupervised Anomaly Detection in Data Quality Control
Authors:
- 15 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2327-2336
Abstract
Data is one of the most valuable assets of an organization and has a tremendous impact on its long-term success and decision-making processes. Typically, organizational data error and outlier detection processes perform manually and reactively, making them time-consuming and prone to human errors. Additionally, rich data types, unlabeled data, and increased volume have made such data more complex. Accordingly, an automated anomaly detection approach is required to improve data management and quality control processes. This study introduces an unsupervised anomaly detection approach based on models comparison, consensus learning, and a combination of rules of thumb with iterative hyper-parameter tuning to increase data quality. Furthermore, a domain expert is considered a human in the loop to evaluate and check the data quality and to judge the output of the unsupervised model. An experiment has been conducted to assess the proposed approach in the context of a case study. The experiment results confirm that the proposed approach can improve the quality of organizational data and facilitate anomaly detection processes.Keywords
This publication has 19 references indexed in Scilit:
- Automating large-scale data quality verificationProceedings of the VLDB Endowment, 2018
- DBSCAN Revisited, RevisitedACM Transactions on Database Systems, 2017
- Open data: Quality over quantityInternational Journal of Information Management, 2017
- Data Quality Management with Semantic Technologies2016
- The Challenges of Data Quality and Data Quality Assessment in the Big Data EraData Science Journal, 2015
- Methodologies for data quality assessment and improvementACM Computing Surveys, 2009
- Special section on workflow systems and applications in e-ScienceFuture Generation Computer Systems, 2009
- A comprehensive data quality methodology for web and structured dataInternational Journal of Innovative Computing and Applications, 2008
- A product perspective on total data quality managementCommunications of the ACM, 1998
- Beyond Accuracy: What Data Quality Means to Data ConsumersJournal of Management Information Systems, 1996