Partitioning techniques for fine-grained indexing
Authors:
Open Access
- 1 April 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636382,p. 1127-1138
Abstract
Many data-intensive websites use databases that grow much faster than the rate that users access the data. Such growing datasets lead to ever-increasing space and performance overheads for maintaining and accessing indexes. Furthermore, there is often considerable skew with popular users and recent data accessed much more frequently. These observations led us to design Shinobi, a system which uses horizontal partitioning as a mechanism for improving query performance to cluster the physical data, and increasing insert performance by only indexing data that is frequently accessed. We present database design algorithms that optimally partition tables, drop indexes from partitions that are infrequently queried, and maintain these partitions as workloads change. We show a 60× performance improvement over traditionally indexed tables using a real-world query workload derived from a traffic monitoring application.Keywords
This publication has 20 references indexed in Scilit:
- CORADDProceedings of the VLDB Endowment, 2010
- TrajStore: An adaptive storage system for very large trajectory data sets2010
- On-Line Index Selection for Shifting Workloads2007
- An Online Approach to Physical Design Tuning2007
- AutoPart: automating schema design for large scientific databases using data partitioning2004
- Integrating vertical and horizontal partitioning into automated physical database design2004
- Write-Optimized B-Trees2004
- Query Optimization in Database SystemsACM Computing Surveys, 1984
- Horizontal data partitioning in database design1982
- Ubiquitous B-TreeACM Computing Surveys, 1979