Cackle: Analytical Workload Cost and Performance Stability With Elastic Pools
Open Access
- 8 December 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 1 (4) , 1-25
Abstract
Analytical query workloads are prone to rapid fluctuations in resource demands. These rapid, hard to predict resource demand changes make provisioning a challenge. Users must either over provision at excessive cost or suffer poor query latency when demand spikes. Prior work shows the viability of using cloud functions to match the supply of compute to the workload demand without provisioning resources ahead of time. For low query volumes, this approach is less costly at reasonable performance compared to provisioned systems, but as query volumes increase the cost overhead of cloud functions outweighs the benefit gained by rapid elasticity. In this work, we propose a novel strategy combining rapidly scalable but expensive resources with slow to start but inexpensive virtual machines to gain the benefit of elasticity without losing out on the cost savings of provisioned resources. We demonstrate a technique that minimizes cost over a wide range of workloads, environmental conditions, and compute costs while providing stable query performance. We implement these ideas in Cackle and demonstrate that it achieves similar performance and cost per query across a wide range of workloads, avoiding the cost and performance cliffs of alternative approaches.Keywords
This publication has 13 references indexed in Scilit:
- Jiffy2022
- The evolution of Amazon redshiftProceedings of the VLDB Endowment, 2021
- Primula2020
- POLARISProceedings of the VLDB Endowment, 2020
- Lambada: Interactive Data Analytics on Cold Data Using Serverless Cloud Infrastructure2020
- Starling: A Scalable Query Engine on Cloud Functions2020
- P-Store2018
- Apache SparkCommunications of the ACM, 2016
- The Snowflake Elastic Data Warehouse2016
- Amazon Redshift and the Case for Simpler Data Warehouses2015