The data cyclotron query processing scheme
Open Access
- 19 December 2011
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Database Systems
- Vol. 36 (4) , 1-35
Abstract
A grand challenge of distributed query processing is to devise a self-organizing architecture which exploits all hardware resources optimally to manage the database hot set, minimize query response time, and maximize throughput without single point global coordination. The Data Cyclotron architecture [Goncalves and Kersten 2010] addresses this challenge using turbulent data movement through a storage ring built from distributed main memory and capitalizing on the functionality offered by modern remote-DMA network facilities. Queries assigned to individual nodes interact with the storage ring by picking up data fragments, which are continuously flowing around, that is, the hot set. The storage ring is steered by the Level Of Interest ( LOI ) attached to each data fragment, which represents the cumulative query interest as it passes around the ring multiple times. A fragment with LOI below a given threshold, inversely proportional to the ring load, is pulled out to free up resources. This threshold is dynamically adjusted in a fully distributed manner based on ring characteristics and locally observed query behavior. It optimizes resource utilization by keeping the average data access latency low. The approach is illustrated using an extensive and validated simulation study. The results underpin the fragment hot set management robustness in turbulent workload scenarios. A fully functional prototype of the proposed architecture has been implemented using modest extensions to MonetDB and runs within a multirack cluster equipped with Infiniband. Extensive experimentation using both microbenchmarks and high-volume workloads based on TPC-H demonstrates its feasibility. The Data Cyclotron architecture and experiments open a new vista for modern distributed database architectures with a plethora of new research challenges.Keywords
This publication has 28 references indexed in Scilit:
- An architecture for recycling intermediates in a column-storeACM Transactions on Database Systems, 2010
- The case for RAMCloudsACM SIGOPS Operating Systems Review, 2010
- Breaking the memory wall in MonetDBCommunications of the ACM, 2008
- Optimizing main-memory join on modern hardwareIEEE Transactions on Knowledge and Data Engineering, 2002
- The state of the art in distributed query processingACM Computing Surveys, 2000
- Towards self-tuning data placement in parallel database systemsACM SIGMOD Record, 2000
- The Datacycle architectureCommunications of the ACM, 1992
- Parallel database systemsCommunications of the ACM, 1992
- The datacycle architecture for very high throughput database systemsACM SIGMOD Record, 1987
- Distributed query processingACM Computing Surveys, 1984