Storm-based distributed sampling system for multi-source stream environment
Open Access
- 20 November 2018
- journal article
- research article
- Published by SAGE Publications in International Journal of Distributed Sensor Networks
- Vol. 14 (11)
Abstract
As a large amount of data streams occur rapidly in many recent applications such as social network service, Internet of Things, and smart factory, sampling techniques have attracted many attentions to handle such data streams efficiently. In this article, we address the performance improvement of binary Bernoulli sampling in the multi-source stream environment. Binary Bernoulli sampling has the n:1 structure where n sites transmit data to 1 coordinator. However, as the number of sites increases or the input stream explosively increases, the binary Bernoulli sampling may cause a severe bottleneck in the coordinator. In addition, bidirectional communication over different networks among the coordinator and sites may incur excessive communication overhead. In this article, we propose a novel distributed processing model of binary Bernoulli sampling to solve these coordinator bottleneck and communication overhead problems. We first present a multiple-coordinator structure to solve the coordinator bottleneck. We then present a new sampling model with an integrated framework and shared memory to alleviate the communication overhead. To verify the effectiveness and scalability of the proposed model, we perform its actual implementation in Apache Storm, a real-time distributed stream processing system. Experimental results show that our Storm-based binary Bernoulli sampling improves performance by up to 1.8 times compared with the legacy method and maintains high performance even when the input stream largely increases. These results indicate that the proposed distributed processing model is an excellent approach that solves the performance degradation problem of binary Bernoulli sampling and verifies its superiority through the actual implementation on Apache Storm.Keywords
This publication has 7 references indexed in Scilit:
- Efficient Two-Step Protocol and Its Discriminative Feature Selections in Secure Similar Document DetectionSecurity and Communication Networks, 2017
- Distributed discovery of frequent subgraphs of a network using MapReduceComputing, 2015
- A Forensic Analysis Method for Redis Database based on RDB and AOF FileJournal of Computers, 2014
- Data mining with big dataIEEE Transactions on Knowledge and Data Engineering, 2013
- Distortion-free predictive streaming time-series matchingInformation Sciences, 2010
- Optimized stratified sampling for approximate query processingACM Transactions on Database Systems, 2007
- Random sampling with a reservoirACM Transactions on Mathematical Software, 1985