Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings
Authors:
Open Access
- 31 May 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 99-113
Abstract
Query processing over big data is ubiquitous in modern clouds, where the system takes care of picking both the physical query execution plans and the resources needed to run those plans, using a cost-based query optimizer. A good cost model, therefore, is akin to better resource efficiency and lower operational costs. Unfortunately, the production workloads at Microsoft show that costs are very complex to model for big data systems. In this work, we investigate two key questions: (i) can we learn accurate cost models for big data systems, and (ii) can we integrate the learned models within the query optimizer. To answer these, we make three core contributions. First, we exploit workload patterns to learn a large number of individual cost models and combine them to achieve high accuracy and coverage over a long period. Second, we propose extensions to Cascades framework to pick optimal resources, i.e, number of containers, during query planning. And third, we integrate the learned cost models within the Cascade-style query optimizer of SCOPE at Microsoft. We evaluate the resulting system, Cleo, in a production environment using both production and TPC-H workloads. Our results show that the learned cost models are 2 to 3 orders of magnitude more accurate, and 20X more correlated with the actual runtimes, with a large majority (70%) of the plan changes leading to substantial improvements in latency as well as resource usage.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Redoop infrastructure for recurring big data queriesProceedings of the VLDB Endowment, 2014
- Advanced join strategies for large-scale distributed computationProceedings of the VLDB Endowment, 2014
- Continuous cloud-scale query optimization and processingProceedings of the VLDB Endowment, 2013
- Robust estimation of resource consumption for SQL queries using statistical techniquesProceedings of the VLDB Endowment, 2012
- SCOPE: parallel databases meet MapReduceThe VLDB Journal, 2012
- Runtime measurements in the cloudProceedings of the VLDB Endowment, 2010
- SCOPEProceedings of the VLDB Endowment, 2008
- Regularization and Variable Selection Via the Elastic NetJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- Stochastic gradient boostingComputational Statistics & Data Analysis, 2002
- New TPC benchmarks for decision support and web commerceACM SIGMOD Record, 2000