Grand Perspective: Load Shedding in Distributed CEP Applications
Preprint
Abstract
In distributed Complex Event Processing (CEP) applications with high load but limited resources, bottleneck operators in the operator graph can significantly slow down processing of event streams, thus compelling the need to shed load. A high-quality load shedding strategy that resolves the bottleneck with high output quality evaluates each event's importance with regards to the application's final output and drops less important events from the event stream for the benefit of important ones. So far, no solution has been proposed that is able to permit good load shedding in distributed, multi-operator CEP applications. On one hand, shedding strategies have been proposed for single-operator CEP applications that can measure an event's importance immediately at the bottleneck operator, only, and thereby ignore the effect of other streams in the application on an event's importance. On the other hand, shedding strategies have been proposed for applications with multiple operators from the area of stream processing that provide a fixed selectivity which is not given in the conditional CEP operators. We, therefore, propose a load-shedding solution for distributed CEP applications that maximizes the application's final output and ensures timely processing of important events by using a set of CEP-tailored selectivity functions and a linear program, which is an abstraction of the CEP application. Moreover, our solution ensures a quality optimal shedder configuration even in the presence of dynamically changing conditions. With the help of extensive evaluations on both synthetic and real data, we show that our solution successfully resolves overload at bottleneck operators and at the same time maximizes the quality of the application's output.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Color-based Lightweight Utility-aware Load Shedding for Real-Time Video Analytics at the Edge2024
- gSPICE: Model-Based Event Shedding in Complex Event Processing2023
- State-Aware Load Shedding From Input Event Streams in Complex Event ProcessingIEEE Transactions on Big Data, 2020
- A Survey on Automatic Parameter Tuning for Big Data Processing SystemsACM Computing Surveys, 2020
- pSPICE: Partial Match Shedding for Complex Event Processing2019
- Pipelined fission for stream programs with dynamic selectivity and partitioned stateJournal of Parallel and Distributed Computing, 2016
- Predictable Low-Latency Event Detection With Parallel Complex Event ProcessingIEEE Internet of Things Journal, 2015
- A catalog of stream processing optimizationsACM Computing Surveys, 2014
- Snoop: An expressive event specification language for active databasesData & Knowledge Engineering, 1994