RCD+: A Partitioning Method for Data Streams Based on Multiple Queries
Open Access
- 13 March 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 52517-52527
Abstract
Big data stream management systems often must transform a query application into multiple query tasks, simultaneously and dynamically partitioning data streams based on attribute values or partitioning keys. However, due to different partitioning orders or strategies of partitioning keys, the redundant and repetitive transmission of data streams at different nodes leads to system performance degradation. In addition, with the change of data skewness, the problem of unbalanced data stream partitioning still exists between different processing units within the same node. This paper presents the partitioning framework RCD+ (Runtime Correlation Discovery) according to runtime correlation discovery. RCD+ implements the full granularity partitioning strategy, which includes runtime positive correlation partitioning (RPC-partitioning) and clustering partitioning (Clu-partitioning). First, in the process of RPC-partitioning, we introduce the mini-batch scheme to reduce the number of output stream caches and partition data streams using the relevance of partitioning keys. Furthermore, in the process of Clu-partitioning, we re-partition data streams by clustering of skewed data streams between the inter-node and the intra-node. Then, we construct the routing table to manage partition states in order to ensure the correctness of multiple query tasks. Finally, we have implemented this framework on Apache Storm. Experiments with synthetic data and real data show that our proposed framework exhibits better query performance.Keywords
Funding Information
- National Natural Science Foundation of China (81703946, 61702164)
- China Postdoctoral Science Foundation (2019M650044)
This publication has 15 references indexed in Scilit:
- Partitioning Road Network Streams Based on Runtime Correlation Discovery2017
- Storm@twitter2014
- Scalable and adaptive online joinsProceedings of the VLDB Endowment, 2014
- High performance stream query processing with correlation-aware partitioningProceedings of the VLDB Endowment, 2013
- Mining interesting locations and travel sequences from GPS trajectories2009
- No pane, no gainACM SIGMOD Record, 2005
- Efficient mining of both positive and negative association rulesACM Transactions on Information Systems, 2004
- Flux: an adaptive partitioning operator for continuous query systems2004
- Streaming-data algorithms for high-quality clustering2003
- Parallel database systemsCommunications of the ACM, 1992