Intelligent Scaling in Amazon Redshift
Authors:
- Vikram NathanVNVikram Nathan
- Vikramank SinghVSVikramank Singh
- Zhengchun LiuZLZhengchun Liu
- Mohammad RahmanMRMohammad Rahman
- Andreas KipfAKAndreas Kipf
- Dominik HornDHDominik Horn
- Davide PaganoDPDavide Pagano
- Gaurav SaxenaGSGaurav Saxena
- Balakrishnan NarayanaswamyBNBalakrishnan Narayanaswamy
- Tim KraskaTKTim Kraska
Open Access
- 9 June 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 269-279
Abstract
Cloud-based data warehouses are built to be easy to use, requiring minimal intervention from customers as their workloads scale. However, there are still many dimensions of a workload that they do not scale with automatically. For example, in cloud-managed clusters, large ad-hoc queries and ETL workloads must use the same cluster size provisioned for the rest of the workload, and warehouse size does not automatically grow as the underlying data grows in size, causing queries to slow down. In this paper, we describe RAIS, the latest collection of AI-powered scaling and optimization techniques in Amazon Redshift, released in preview at re:Invent 2023, which enable it to scale both vertically and horizontally to adapt to all types of workload variability. RAIS dynamically provisions compute resources to run heavy queries efficiently and automatically optimizes warehouse size for the customer's workload, even as it shifts over time. We show that, depending on the workload, RAIS improves either cost or average query execution time by up to 7.6× and 14.2×, respectively, over existing baselines.Keywords
This publication has 12 references indexed in Scilit:
- Auto-WLM: Machine Learning Enhanced Workload Management in Amazon Redshift2023
- LSched: A Workload-Aware Learned Query Scheduler for Analytical Database Systems2022
- Amazon Redshift Re-invented2022
- Bao: Making Learned Query Optimization PracticalACM SIGMOD Record, 2021
- Learning scheduling algorithms for data processing clusters2019
- Automatic Database Management System Tuning Through Large-scale Machine Learning2017
- WiSeDBProceedings of the VLDB Endowment, 2016
- iCBSProceedings of the VLDB Endowment, 2011
- SLA-tree2011
- Tuning database configuration parameters with iTunedProceedings of the VLDB Endowment, 2009