Fast Record Linkage for Company Entities
Authors:
- 1 December 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 623-630
Abstract
Record linkage is an essential part of nearly all real-world systems that consume structured and unstructured data coming from different sources. Typically no common key is available for connecting records. Massive data integration processes often have to be completed before any data analytics and further processing can be performed. In this work we focus on company entity matching, where company name, location and industry are taken into account. Our contribution is a highly scalable, enterprise-grade end-to-end system that uses rule-based linkage algorithms in combination with a machine learning approach to account for short company names. Linkage time is greatly reduced by an efficient decomposition of the search space using MinHash. Based on real-world ground truth datasets, we show that our approach reaches a recall of 91% compared to 73% for baseline approaches, while scaling linearly with the number of nodes used in the system.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Deep Learning for Entity Matching2018
- Active Learning for Large-Scale Entity Resolution2017
- MagellanProceedings of the VLDB Endowment, 2016
- Entity Resolution Using Convolutional Neural NetworkProcedia Computer Science, 2016
- Mining of Massive Datasets2014
- The effect of data cleaning on record linkage qualityBMC Medical Informatics and Decision Making, 2013
- Entity resolutionProceedings of the VLDB Endowment, 2012
- Similarity estimation techniques from rounding algorithms2002
- Min-wise independent permutations (extended abstract)1998
- A Theory for Record LinkageJournal of the American Statistical Association, 1969