Filtering and Accelerating: A Unified Framework for High-Performance Persistence Estimation
- 22 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 38 (8) , 5095-5112
Abstract
Efficient data stream processing, particularly for persistence estimation, is crucial in handling high-velocity data streams characterized by skewed distributions of item frequencies. Unlike more straightforward frequency metrics, persistence captures items' recurrence across multiple time windows, posing a significant challenge to existing single-structure sketches where high-persistence and low-persistence items collide. To address this, we introduce the Hypersistent Sketch, a unified framework for high-performance estimation built on two decoupled mechanisms: filtering and accelerating. The filtering component, a Cold Filter, directly addresses the skewed nature of data streams. It separates hot items from the majority of cold ones, which allows for differential treatment. The accelerating component, a Burst Filter, then optimizes the processing of hot items. It significantly improves throughput by preventing repeated insertions within a single window. We demonstrate its generality by applying it to various state-of-the-art sketches (e.g., On-Off, Waving, P-Sketch), showing it consistently enhances their original performance. We also deploy our framework on Redis platforms, demonstrating the framework’s broad applicability and scalability.Keywords
This publication has 52 references indexed in Scilit:
- BitMatcher: Bit-level Counter Adjustment for Sketches2024
- CBA Sketch: A Sketching Algorithm Mining Persistent Batches in Data Streams2024
- Advanced Persistent Threats (APT): evolution, anatomy, attribution and countermeasuresJournal of Ambient Intelligence and Humanized Computing, 2023
- Discovering unknown advanced persistent threat using shared features mined by neural networksComputer Networks, 2021
- Finding needles in a hay stream: On persistent item lookup in data streamsComputer Networks, 2020
- Resource allocation and computation offloading with data security for mobile edge computingFuture Generation Computer Systems, 2019
- Finding Significant Items in Data Streams2019
- A Survey on Advanced Persistent Threats: Techniques, Solutions, Challenges, and Research OpportunitiesIEEE Communications Surveys & Tutorials, 2019
- Detecting Advanced Persistent Threats Based on Entropy and Support Vector Machine2018
- Monitoring persistent items in the union of distributed streamsJournal of Parallel and Distributed Computing, 2014