Dejan Škvorc

Data stream processing systems are used to continuously run mission-critical applications for real-time monitoring and alerting. These systems require high throughput and low latency to process incoming data streams in real time. However, changes in the distribution of incoming data streams over time can cause partition skew, which is defined as an unequal distribution of data partitions among workers, resulting in sub-optimal processing due to an unbalanced load. This paper presents the first solution designed specifically to address partition skew in the context of joining streaming and static data. Our solution uses state-of-the-art principles to monitor processing load, detect load imbalance, and dynamically redistribute partitions, to achieve optimal load balance. To accomplish this, our solution leverages the collocation of streaming and static data, while considering the processing load of the join and the subsequent stream processing operations. Finally, we present the results of an experimental evaluation, in which we compared the throughput and latency of four stream processing pipelines containing such a join. The results show that our solution achieved significantly higher throughput and lower latency than the competing approaches.

To adapt to rapidly changing market conditions and increase the return of investment, today's IT solutions usually combine service-oriented architecture (SOA) and event-driven architecture (EDA) that support reusability, flexibility, and responsiveness of business processes. Programming languages for development of event-driven service compositions face several main challenges. First, a language should be based on standard service composition languages to be compatible with SOA-enabling technologies. Second, a language should enable seamless integration of services into event-driven workflows. Third, to overcome a knowledge divide, language should enable seamless cooperation between application developers with different skills and knowledge. Since WS-BPEL is widely accepted as standard executable language in SOA, we extended WS-BPEL with support for event-driven workflow coordination. We designed event-handling mechanisms as special-purpose Coopetition services and augmented WS-BPEL with primitives for their invocation. Coopetition services augment SOA with fundamental EDA characteristics: decoupled interactions, many-to-many communication, publish/subscribe messaging, event triggering, and asynchronous operations. To make the application development familiar to wide community of developers, we designed an application-level end-user language on top of WS-BPEL whose primitives for invocation of regular Web services and Coopetition services resemble the constructs of typical scripting and coordination language.