Fast and Scalable Data Transfer Across Data Systems
Open Access
- 17 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 3 (3) , 1-28
Abstract
Fast and scalable data transfer is crucial in today's decentralized data ecosystems and data-driven applications. Example use cases include transferring data from operational systems to consolidated data warehouse environments, or from relational database systems to data lakes for exploratory data analysis or ML model training. Traditional data transfer approaches rely on efficient point-to-point connectors or general middleware with generic intermediate data representations. Physical environments (e.g., on-premise, cloud, or consumer nodes) also have become increasingly heterogeneous. Existing work still struggles to achieve both, fast and scalable data transfer as well as generality in terms of heterogeneous systems and environments. Hence, in this paper, we introduce a holistic data transfer framework. Our XDBC framework splits the data transfer pipeline into logical components and provides a wide variety of physical implementations for these components. This design allows a seamless integration of different systems as well as the automatic optimizations of data transfer configurations according to workload and environment characteristics. Our evaluation shows that XDBC outperforms state-of-the-art generic data transfer tools by up to 5x, while being on par with specialized approaches.Keywords
Funding Information
- German Federal Ministry of Education and Research (BIFOLD24B)
- German Federal Ministry of Education and Research (01IS17052)
This publication has 49 references indexed in Scilit:
- SheetReader: Efficient Specialized Spreadsheet ParsingInformation Systems, 2023
- Optimizing Cross-Platform Data Movement2019
- Presto: SQL on Everything2019
- Muses: Distributed Data Migration System for Polystores2019
- Data transformation and migration in polystores2016
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- An Overview of SQL Support in Workflow Products2008
- Optimizing ETL Processes in Data Warehouses2005
- Integrating Association Rule Mining with Relational Database Systems: Alternatives and ImplicationsData Mining and Knowledge Discovery, 2000
- Twenty Five Years of Cyclic Queues and Closed Queue Networks: A ReviewJournal of the Operational Research Society, 1982