Adaptive Product Normalization: Using Online Learning for Record Linkage in Comparison Shopping
Authors:
- 5 January 2006
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 58-65
Abstract
The problem of record linkage focuses on determining whether two object descriptions refer to the same underlying entity. Addressing this problem effectively has many practical applications, e.g., elimination of duplicate records in databases and citation matching for scholarly articles. In this paper, we consider a new domain where the record linkage problem is manifested: Internet comparison shopping. We address the resulting linkage setting that requires learning a similarity function between record pairs from streaming data. The learned similarity function is subsequently used in clustering to determine which records are co-referent and should be linked. We present an online machine learning method for addressing this problem, where a composite similarity function based on a linear combination of basis functions is learned incrementally. We illustrate the efficacy of this approach on several real-world datasets from an Internet comparison shopping site, and show that our method is able to effectively learn various distance functions for product data with differing characteristics. We also provide experimental results that show the importance of considering multiple performance measures in record linkage evaluation.Keywords
This publication has 20 references indexed in Scilit:
- A Bayesian decision model for cost optimal record matchingThe VLDB Journal, 2003
- Efficient clustering of high-dimensional data sets with application to reference matching2000
- Large Margin Classification Using the Perceptron AlgorithmMachine Learning, 1999
- Data clusteringACM Computing Surveys, 1999
- Autonomous citation matching1999
- Algorithms on Strings, Trees and Sequences1997
- A scalable comparison-shopping agent for the World-Wide Web1997
- The merge/purge problem for large databases1995
- Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, FloridaJournal of the American Statistical Association, 1989
- A Theory for Record LinkageJournal of the American Statistical Association, 1969