Preprint
Abstract
Cloud data centers are evolving fast. At the same time, today's large-scale data analytics applications require non-trivial performance tuning that is often specific to the applications, workloads, and data center infrastructure. We propose TeShu, which makes network shuffling an extensible unified service layer common to all data analytics. Since an optimal shuffle depends on a myriad of factors, TeShu introduces parameterized shuffle templates, instantiated by accurate and efficient sampling that enables TeShu to dynamically adapt to different application workloads and data center layouts. Our preliminary experimental results show that TeShu efficiently enables shuffling optimizations that improve performance and adapt to a variety of data center network scenarios.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Cheetah: Accelerating Database Queries with Switch Pruning2020
- Starling: A Scalable Query Engine on Cloud Functions2020
- Optimizing Declarative Graph Queries at Large Scale2019
- Big Data Analytics with Datalog Queries on Spark2016
- Towards Memory-Optimized Data Shuffling Patterns for Big Data Analytics2016
- Accelerating Data Shuffling in MapReduce Framework with Scale-up NUMA Computing Architecture2016
- Supporting scalable analytics with latency constraintsProceedings of the VLDB Endowment, 2015
- Storm@twitter2014
- Symbiotic routing in future data centersACM SIGCOMM Computer Communication Review, 2010
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005