Incremental, iterative data processing with timely dataflow
Authors:
- Frank McSherryFMFrank McSherry
Open Access
- 22 September 2016
- journal article
- Published by Association for Computing Machinery (ACM) in Communications of the ACM
- Vol. 59 (10) , 75-83
Abstract
We describe the timely dataflow model for distributed computation and its implementation in the Naiad system. The model supports stateful iterative and incremental computations. It enables both low-latency stream processing and high-throughput batch processing, using a new approach to coordination that combines asynchronous and fine-grained synchronous execution. We describe two of the programming frameworks built on Naiad: GraphLINQ for parallel graph processing, and differential dataflow for nested iterative and incremental computations. We show that a general-purpose system can achieve performance that matches, and sometimes exceeds, that of specialized systems.Keywords
This publication has 15 references indexed in Scilit:
- Timely Rollback: Specification and Verification2015
- MillWheelProceedings of the VLDB Endowment, 2013
- Spinning fast iterative data flowsProceedings of the VLDB Endowment, 2012
- DremelProceedings of the VLDB Endowment, 2010
- On-the-fly progress detection in iterative stream queriesProceedings of the VLDB Endowment, 2009
- MapReduceCommunications of the ACM, 2008
- Dryad2007
- Exploiting punctuation semantics in continuous data streamsIEEE Transactions on Knowledge and Data Engineering, 2003
- The derivation of distributed termination detection algorithms from garbage collection schemesACM Transactions on Programming Languages and Systems, 1993
- Parallel database systemsCommunications of the ACM, 1992