Duplicate record detection approach based on sentence embeddings
Authors:
- 1 September 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 269-274
Abstract
Duplicate record detection is a crucial task for data cleaning. Records representation is among the main challenges of this task. Word embeddings models have been widely applied in an attempt to improve records representation. However, despite the improvements made by word embeddings to enhance the semantic aspect, duplicate record detection results is still insufficient In this paper, we present a duplicate record detection approach based on sentence embeddings, where each attribute is viewed as a sentence. First, universal sentence encoder model is used to embed the values of records’ attributes into embeddings vectors. Afterwards, based on the created vectors, similarity vectors between the record pairs are computed. Finally, support vector machine algorithm is used to classify the similarity vectors. Experiments on two datasets (Cora and Restaurant) show that our proposal outperforms state-of-the-art baselines and leads to significant improvements in duplicate record detection effectiveness.Keywords
This publication has 25 references indexed in Scilit:
- Data QualityACM SIGMOD Record, 2015
- Deep learningNature, 2015
- Deep Unordered Composition Rivals Syntactic Methods for Text Classification2015
- Two/Too Simple Adaptations of Word2Vec for Syntax Problems2015
- Glove: Global Vectors for Word Representation2014
- Merging the Results of Approximate Match Operations2004
- Learning domain-independent string transformation weights for high accuracy object identification2002
- Interactive deduplication using active learning2002
- Text categorization with Support Vector Machines: Learning with many relevant features1998
- Identification of common molecular subsequencesJournal of Molecular Biology, 1981