Safe Shared State in Dataflow Systems
Open Access
- 24 June 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 30-41
Abstract
The complexity of implementing scalable data processing tasks in parallel and distributed computing environments has pushed the adoption of restricted programming models that simplify expressing such tasks. In this context, the dataflow model has emerged as the standard for building distributed data processing systems. The key insight of this model is to express computations as sequences of operations that do not share any state and can thus be deployed and executed independently on the same or on different machines. However, entirely excluding state sharing may be detrimental for performance, as it prevents operations executed on the same machine from accessing common resources. The effects of these limitations become more and more evident as the computational and memory resources of individual hosts increase, and even large-scale data analysis tasks can be performed with few well-equipped machines. Moving from this observation, in this paper we present an extension to the classic dataflow model that enables disciplined state sharing across operators to improve performance. We implement the model in Renoir, a dataflow system written in Rust, and we exploit the characteristics of this programming language to ensure safe access to shared state by design. We present several use cases where our model may be beneficial, and we use them to evaluate the proposed model in terms of performance and code complexity. We show that our model empowers developers to better exploit the resources of machines, resulting in substantial performance improvements in several use cases. Programs written with our extended model may even be more concise and simpler than in the classic dataflow model.Keywords
Funding Information
- National Recovery and Resilience Plan (NRRP), Mission 4 Component 2 Investment 1.3 - Call for tender No. 1561 of 11.10.2022 of Ministero dellðUni- versità e della Ricerca (MUR); funded by the European Union ð NextGenerationEU (Project code PE0000021, Concession Decree No. 1561 of 11.10.2022 adopted by Ministero dell'Università e della Ricerca (MUR), CUP - to be indicated by each Benefi- ciary, according to attachment E of Decree No. 1561/2022)
This publication has 25 references indexed in Scilit:
- The Renoir Dataflow Platform: Efficient Data Processing without ComplexityFuture Generation Computer Systems, 2024
- RustHornBelt: a semantic foundation for functional verification of Rust programs with unsafe code2022
- Distributed transactions on serverless stateful functions2021
- Safe systems programming in RustCommunications of the ACM, 2021
- TSpoon: Transactions on a stream processorJournal of Parallel and Distributed Computing, 2020
- A survey of state management in big data processing systemsThe VLDB Journal, 2018
- FlowDB2017
- Apache SparkCommunications of the ACM, 2016
- Twitter Heron2015
- MapReduceCommunications of the ACM, 2008