Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures
Preprint
Abstract
Distributed stream processing engines are designed with a focus on scalability to process big data volumes in a continuous manner. We present the Theodolite method for benchmarking the scalability of distributed stream processing engines. Core of this method is the definition of use cases that microservices implementing stream processing have to fulfill. For each use case, our method identifies relevant workload dimensions that might affect the scalability of a use case. We propose to design one benchmark per use case and relevant workload dimension. We present a general benchmarking framework, which can be applied to execute the individual benchmarks for a given use case and workload dimension. Our framework executes an implementation of the use case's dataflow architecture for different workloads of the given dimension and various numbers of processing instances. This way, it identifies how resources demand evolves with increasing workloads. Within the scope of this paper, we present 4 identified use cases, derived from processing Industrial Internet of Things data, and 7 corresponding workload dimensions. We provide implementations of 4 benchmarks with Kafka Streams and Apache Flink as well as an implementation of our benchmarking framework to execute scalability benchmarks in cloud environments. We use both for evaluating the Theodolite method and for benchmarking Kafka Streams' and Flink's scalability for different deployment options.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- Goals and measures for analyzing power consumption data in manufacturing enterprisesJournal of Data, Information and Management, 2021
- Performance evaluation of real-time stream processing systems for Internet of Things applicationsFuture Generation Computer Systems, 2020
- Scalable and Reliable Multi-dimensional Sensor Data Aggregation in Data Streaming ArchitecturesData-Enabled Discovery and Applications, 2020
- Towards a Framework for Data Stream Processing in the FogInformatik Spektrum, 2019
- Stream Window Aggregation Semantics and Optimization2019
- Software Architecture: Past, Present, Future2018
- Senska – Towards an Enterprise Streaming Benchmark2017
- RIoTBench: An IoT benchmark for distributed stream processing systemsConcurrency and Computation: Practice and Experience, 2017
- Cloud Service Benchmarking2017
- WESSBAS: extraction of probabilistic workload specifications for load testing and performance prediction—a model-driven approach for session-based application systemsSoftware and Systems Modeling, 2016