KerA: Scalable Data Ingestion for Stream Processing
Authors:
- Ovidiu-Cristian MarcuOMOvidiu-Cristian Marcu
- Alexandru CostanACAlexandru Costan
- Gabriel AntoniuGAGabriel Antoniu
- Maria Perez-HernandezMPMaria Perez-Hernandez
- Bogdan NicolaeBNBogdan Nicolae
- Radu TudoranRTRadu Tudoran
- Stefano BortoliSBStefano Bortoli
- 1 July 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1480-1485
Abstract
Big Data applications are increasingly moving from batch-oriented execution models to stream-based models that enable them to extract valuable insights close to real-time. To support this model, an essential part of the streaming processing pipeline is data ingestion, i.e., the collection of data from various sources (sensors, NoSQL stores, filesystems, etc.) and their delivery for processing. Data ingestion needs to support high throughput, low latency and must scale to a large number of both data producers and consumers. Since the overall performance of the whole stream processing pipeline is limited by that of the ingestion phase, it is critical to satisfy these performance goals. However, state-of-art data ingestion systems such as Apache Kafka build on static stream partitioning and offset-based record access, trading performance for design simplicity. In this paper we propose KerA, a data ingestion framework that alleviate the limitations of state-of-art thanks to a dynamic partitioning scheme and to lightweight indexing, thereby improving throughput, latency and scalability. Experimental evaluations show that KerA outperforms Kafka up to 4x for ingestion throughput and up to 5x for the overall stream processing throughput. Furthermore, they show that KerA is capable of delivering data fast enough to saturate the big data engine acting as the consumer.Keywords
This publication has 7 references indexed in Scilit:
- Towards a unified storage and ingestion architecture for stream processing2017
- Drizzle2017
- DistributedLog: A High Performance Replicated Log Service2017
- Implementing linearizability at large scale and low latency2015
- The RAMCloud Storage SystemACM Transactions on Computer Systems, 2015
- Durability with BookKeeperACM SIGOPS Operating Systems Review, 2013
- Fast crash recovery in RAMCloud2011