Gen-T: Table Reclamation in Data Lakes
Authors:
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3532-3545
Abstract
We introduce the problem of Table Reclamation. Given a Source Table and a large table repository, reclamation finds a set of tables that, when integrated, reproduce the source table as closely as possible. Unlike query discovery problems like Query-by-Example or by-Target, Table Reclamation focuses on reclaiming the data in the Source Table as fully as possible using real tables that may be incomplete or inconsistent. To do this, we define a new measure of table similarity, called error-aware instance similarity, to measure how close a reclaimed table is to a Source Table, a measure grounded in instance similarity used in data exchange. Our search covers not only Select-project-join queries, but integration queries with unions, outerjoins, and the unary operators subsumption and complementation that have been shown to be important in data integration and fusion. Using reclamation, a data scientist can understand if any tables in a repository can be used to exactly reclaim a tuple in the Source. If not, one can understand if this is due to differences in values or to incompleteness in the data. Our solution, Gen-T, performs table discovery to retrieve a set of candidate tables from the table repository, filters these down to a set of originating tables, then integrates these tables to reclaim the Source as closely as possible. We show that our solution, while approximate, is accurate, efficient and scalable in the size of the table repository with experiments on real data lakes containing up to 15K tables, where the average number of tuples varies from small (web tables) to extremely large (open data tables) up to 1M tuples.Keywords
All Related Versions
Funding Information
- NSF (IIS-2107248,IIS-1956096,IIS-2325632)
This publication has 69 references indexed in Scilit:
- P2KG: Declarative Construction and Quality Evaluation of Knowledge Graph from Polystores2023
- Holistic primary key and foreign key detectionJournal of Intelligent Information Systems, 2019
- Provenance in Databases2018
- The Mannheim Search Join EngineJournal of Web Semantics, 2015
- MapMerge: correlating independent schema mappingsThe VLDB Journal, 2012
- Introduction to Information Retrieval2008
- Data exchange: semantics and query answeringTheoretical Computer Science, 2005
- Elements of Information Theory2005
- COMA — A system for flexible combination of schema matching approaches2002
- A survey of approaches to automatic schema matchingThe VLDB Journal, 2001