Resource Management and Scheduling in Distributed Stream Processing Systems
Authors:
Open Access
- 28 May 2020
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 53 (3) , 1-41
Abstract
Stream processing is an emerging paradigm to handle data streams upon arrival, powering latency-critical application such as fraud detection, algorithmic trading, and health surveillance. Though there are a variety of Distributed Stream Processing Systems (DSPSs) that facilitate the development of streaming applications, resource management and task scheduling is not automatically handled by the DSPS middleware and requires a laborious process to tune toward specific deployment targets. As the advent of cloud computing has supported renting resources on-demand, it is of great interest to review the research progress of hosting streaming systems in clouds under certain Service Level Agreements (SLA) and cost constraints. In this article, we introduce the hierarchical structure of streaming systems, define the scope of the resource management problem, and present a comprehensive taxonomy in this context covering critical research topics such as resource provisioning, operator parallelisation, and task scheduling. The literature is then reviewed following the taxonomy structure, facilitating a deeper understanding of the research landscape through classification and comparison of existing works. Finally, we discuss the open issues and future research directions toward realising an automatic, SLA-aware resource management framework.Keywords
This publication has 141 references indexed in Scilit:
- Partitioning functions for stateful data parallelism in stream processingThe VLDB Journal, 2013
- Elastic stream processing in the CloudWIREs Data Mining and Knowledge Discovery, 2013
- A framework for partitioning and execution of data stream applications in mobile cloud computingACM SIGMETRICS Performance Evaluation Review, 2013
- Processing flows of informationACM Computing Surveys, 2012
- Resource allocation for multiple concurrent in-network stream-processing applicationsParallel Computing, 2011
- LimeACM SIGPLAN Notices, 2010
- COLA: Optimizing Stream Processing Applications via Graph Partitioning2009
- Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System2006
- VivaldiACM SIGCOMM Computer Communication Review, 2004
- The vision of autonomic computingComputer, 2003