GeaFlow: A Graph Extended and Accelerated Dataflow System
Open Access
- 13 June 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 1 (2) , 1-27
Abstract
GeaFlow is a distributed dataflow system optimized for streaming graph processing, and has been widely adopted at Ant Group, serving various scenarios ranging from risk control of financial activities to analytics on social networks and knowledge graphs. It is built on top of a base with full-fledged stateful stream processing capabilities, extended with a series of graph-aware optimizations to address the space explosion and programming complexity issues of conventional join-based approaches. We propose new state backends and streaming operators that facilitate processing on dynamic graph-structured datasets, reducing space consumed by states. We develop a hybrid domain-specific language that embeds Gremlin into SQL, supporting both table and graph abstractions over streaming data. In addition to streaming workloads, GeaFlow is also extensively used for some batch processing jobs. In the largest deployments to date, GeaFlow is able to process tens of millions of events per second and manage hundreds of terabytes of states.Keywords
This publication has 55 references indexed in Scilit:
- ImmortalGraphACM Transactions on Storage, 2015
- GraphMatProceedings of the VLDB Endowment, 2015
- TencentRec2015
- LLAMA: Efficient graph analytics using Large Multiversioned Arrays2015
- A catalog of stream processing optimizationsACM Computing Surveys, 2014
- Discretized streams2013
- X-Stream2013
- MillWheelProceedings of the VLDB Endowment, 2013
- Kineograph2012
- Efficiently compiling efficient query plans for modern hardwareProceedings of the VLDB Endowment, 2011