Abstract
The complexity of implementing scalable data processing tasks in parallel and distributed computing environments has pushed the adoption of restricted programming models that simplify expressing such tasks. In this context, the dataflow model has emerged as the standard for building distributed data processing systems. The key insight of this model is to express computations as sequences of operations that do not share any state and can thus be deployed and executed independently on the same or on different machines. However, entirely excluding state sharing may be detrimental for performance, as it prevents operations executed on the same machine from accessing common resources. The effects of these limitations become more and more evident as the computational and memory resources of individual hosts increase, and even large-scale data analysis tasks can be performed with few well-equipped machines. Moving from this observation, in this paper we present an extension to the classic dataflow model that enables disciplined state sharing across operators to improve performance. We implement the model in Renoir, a dataflow system written in Rust, and we exploit the characteristics of this programming language to ensure safe access to shared state by design. We present several use cases where our model may be beneficial, and we use them to evaluate the proposed model in terms of performance and code complexity. We show that our model empowers developers to better exploit the resources of machines, resulting in substantial performance improvements in several use cases. Programs written with our extended model may even be more concise and simpler than in the classic dataflow model.
Funding Information
  • National Recovery and Resilience Plan (NRRP), Mission 4 Component 2 Investment 1.3 - Call for tender No. 1561 of 11.10.2022 of Ministero dellðUni- versità e della Ricerca (MUR); funded by the European Union ð NextGenerationEU (Project code PE0000021, Concession Decree No. 1561 of 11.10.2022 adopted by Ministero dell'Università e della Ricerca (MUR), CUP - to be indicated by each Benefi- ciary, according to attachment E of Decree No. 1561/2022)

This publication has 25 references indexed in Scilit: