Muses: Distributed Data Migration System for Polystores
Authors:
- 1 April 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1602-1605
Abstract
Large datasets can originate from various sources and are being stored in heterogeneous formats, schemas, and locations. Typical data science tasks need to combine those datasets in order to increase their value and extract knowledge. This is done in various data processing systems with diverse execution engines. In order to take advantage of each execution engine's characteristics and APIs data scientists need to migrate and transform their datasets at a very high computational cost and manual labor. Data migration is challenging for two main reasons: i) execution engines expect specific types/shapes of the data as input; ii) there are various physical representations of the data (e.g., partitions). Therefore, migrating data efficiently requires knowledge of systems internals and assumptions. In this paper we present Muses, a distributed, high-performance data migration engine that is able to forward, transform, repartition, and broadcast data between distributed engines' instances efficiently. Muses does not require any changes in the underlying execution engines. In an experimental evaluation, we show that migrating data from one execution engine to another (in order to take advantage of faster, native operations) can increase a pipeline's performance by 30%.Keywords
This publication has 9 references indexed in Scilit:
- Evaluating Genomic Big Data Operations on SciDB and Spark2017
- Data transformation and migration in polystores2016
- Cross-engine query execution in federated database systems2016
- PipeGen2016
- Framework for Supporting Genomic OperationsIEEE Transactions on Computers, 2016
- Apache Flink in current researchit - Information Technology, 2016
- The BigDAWG Polystore SystemACM SIGMOD Record, 2015
- The Stratosphere platform for big data analyticsThe VLDB Journal, 2014
- CARTILAGE2013