Enabling Real-Time Querying of Live and Historical Stream Data
Authors:
- 1 July 2007
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in 19th International Conference on Scientific and Statistical Database Management (SSDBM 2007)
- No. 15516393,p. 28
Abstract
Applications that query data streams in order to identify trends, patterns, or anomalies can often benefit from comparing the live stream data with archived historical stream data. However, searching this historical data in real time has been considered so far to be prohibitively expensive. One of the main bottlenecks is the update costs of the indices over the archived data. In this paper, we address this problem by using our highly-efficient bitmap indexing technology (called FastBit) and demonstrate that the index update operations are sufficiently efficient for this bottleneck to be removed. We describe our prototype system based on the TelegraphCQ streaming query processor and the FastBit bitmap index. We present a detailed performance evaluation of our system using a complex query workload for analyzing real network traffic data. The combined system uses TelegraphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We demonstrate that our system can simultaneously analyze (1) live streams with high data rates and (2) a large repository of historical stream data.Keywords
This publication has 20 references indexed in Scilit:
- Optimizing bitmap indices with efficient compressionACM Transactions on Database Systems, 2006
- Declarative Network Monitoring with an Underprovisioned Query Processor2006
- Building a time machine for efficient recording and retrieval of high-volume network traffic2005
- Using Data Stream Management Systems for Traffic Analysis – A Case Study –Lecture Notes in Computer Science, 2004
- On the Performance of Bitmap Indices for High Cardinality Attributes12004
- Remembrance of Streams Past2004
- Issues in data stream managementACM SIGMOD Record, 2003
- Ripple joins for online aggregation1999
- An efficient bitmap encoding scheme for selection queriesACM SIGMOD Record, 1999
- Improved query performance with variant indexes1997