Adaptive Fuzzy String Matching: How to Merge Datasets with Only One (Messy) Identifying Field
Open Access
- 11 October 2021
- journal article
- letter
- Published by Cambridge University Press (CUP) in Political Analysis
- Vol. 30 (4) , 590-596
Abstract
A single dataset is rarely sufficient to address a question of substantive interest. Instead, most applied data analysis combines data from multiple sources. Very rarely do two datasets contain the same identifiers with which to merge datasets; fields like name, address, and phone number may be entered incorrectly, missing, or in dissimilar formats. Combining multiple datasets absent a unique identifier that unambiguously connects entries is called the record linkage problem. While recent work has made great progress in the case where there are many possible fields on which to match, the much more uncertain case of only one identifying field remains unsolved: this fuzzy string matching problem, both its own problem and a component of standard record linkage problems, is our focus. We design and validate an algorithmic solution called Adaptive Fuzzy String Matching rooted in adaptive learning, and show that our tool identifies more matches, with higher precision, than existing solutions. Finally, we illustrate its validity and practical value through applications to matching organizations, places, and individuals.Keywords
This publication has 20 references indexed in Scilit:
- Mapping the Ideological MarketplaceAmerican Journal of Political Science, 2013
- Quality Over Quantity: Amici Influence and Judicial Decision MakingAmerican Political Science Review, 2013
- Probabilistic Data Generation for Deduplication and Data Linkage2005
- A Panel Probit Analysis of Campaign Contributions and Roll-Call VotesAmerican Journal of Political Science, 2001
- Iterative Automated Record Linkage Using Mixture ModelsJournal of the American Statistical Association, 2001
- Learning string-edit distanceIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998
- Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, FloridaJournal of the American Statistical Association, 1989
- User Centered System Design: New Perspectives on Human-Computer InteractionThe American Journal of Psychology, 1988
- Approximate String MatchingACM Computing Surveys, 1980
- A Theory for Record LinkageJournal of the American Statistical Association, 1969