SimClone: Detecting Tabular Data Clones Using Value Similarity
Open Access
- 27 December 2024
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 34 (1) , 1-27
Abstract
Data clones are defined as multiple copies of the same data among datasets. The presence of data clones between datasets can cause issues such as difficulties in managing data assets and data license violations when using datasets with clones to build AI software. However, detecting data clones is not trivial. The majority of the prior studies in this area rely on structural information to detect data clones (e.g., font size, column header). However, tabular datasets used to build AI software are typically stored without any structural information. In this article, we propose a novel method called SimClone for data clone detection in tabular datasets without relying on structural information. SimClone method utilizes value similarities for data clone detection. We also propose a visualization approach as a part of our SimClone method to help locate the exact position of the cloned data between a dataset pair. Our results show that our SimClone outperforms the current state-of-the-art method by at least 20% in terms of both F1-score and AUC. In addition, SimClone’s visualization component helps identify the exact location of the data clone in a dataset with a Precision@10 value of 0.80 in the top 20 true positive predictions.Keywords
All Related Versions
This publication has 55 references indexed in Scilit:
- Tabular data: Deep learning is not all you needInformation Fusion, 2022
- Data set quality in Machine Learning: Consistency measure based on Group Decision MakingApplied Soft Computing, 2021
- A comparative analysis of gradient boosting algorithmsArtificial Intelligence Review, 2020
- Graph neural networks: A review of methods and applicationsAI Open, 2020
- Big Data Quality Assurance Through Data Traceability: A Case Study of the National Standard Reference Data Program of KoreaIEEE Access, 2019
- Machine learning on big data: Opportunities and challengesNeurocomputing, 2017
- A random forest guided tourTEST, 2016
- A survey on data leakage prevention systemsJournal of Network and Computer Applications, 2016
- Enron's Spreadsheets and Related Emails: A Dataset and Analysis2015
- Data clone detection and visualization in spreadsheets2013