Zero-Shot Cost Models for Parallel Stream Processing
Open Access
- 18 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Abstract
This paper addresses the challenge of predicting the level of parallelism in distributed stream processing (DSP) systems, which are essential to deal with different high workload requirements of various industries such as e-commerce, online gaming, etc., where DSP systems are extensively used. Existing DSP systems rely on either manual tuning of parallelism degree or workload-driven learned models for tuning parallelism, which is either not efficient or can lead to costly operator migrations and downtime when there are workload drifts. Thus, we argue for a learned model that can autonomously decide on the right parallelism degree while generalizing across workloads and meeting the current demands of DSP applications. We propose a novel approach that leverages zero-shot cost models to predict parallelism degree while generalizing across unseen streaming workloads out-of-the-box. To reduce training effort, we propose a rule-based strategy that selects parallelism degree and meaningful transferable features related to query workload and hardware that influences the parallelism decisions. We demonstrate the effectiveness of our strategy by evaluating it with different amount of training queries and show that it achieves lower costs for parallel continuous query processing.Keywords
This publication has 18 references indexed in Scilit:
- Resource Management and Scheduling in Distributed Stream Processing SystemsACM Computing Surveys, 2020
- Managing the Complexity of Processing Financial Data at Scale - An Experience Report2019
- A Comprehensive Survey on Parallelization and Elasticity in Stream ProcessingACM Computing Surveys, 2019
- Decentralized self-adaptation for elastic Data Stream ProcessingFuture Generation Computer Systems, 2018
- State management in Apache Flink®Proceedings of the VLDB Endowment, 2017
- Big data analytics on Apache SparkInternational Journal of Data Science and Analytics, 2016
- SABER2016
- Twitter Heron2015
- Predictable Low-Latency Event Detection With Parallel Complex Event ProcessingIEEE Internet of Things Journal, 2015
- Latency-aware elastic scaling for distributed data stream processing systems2014