Mantle
Authors:
Open Access
- 15 November 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Abstract
Migrating resources is a useful tool for balancing load in a distributed system, but it is difficult to determine when to move resources, where to move resources, and how much of them to move. We look at resource migration for file system metadata and show how CephFS's dynamic subtree partitioning approach can exploit varying degrees of locality and balance because it can partition the namespace into variable sized units. Unfortunately, the current metadata balancer is complicated and difficult to control because it struggles to address many of the general resource migration challenges inherent to the metadata management problem. To help decouple policy from mechanism, we introduce a programmable storage system that lets the designer inject custom balancing logic. We show the flexibility and transparency of this approach by replicating the strategy of a state-of-the-art metadata balancer and conclude by comparing this strategy to other custom balancers on the same system.Keywords
This publication has 14 references indexed in Scilit:
- Towards Dynamic Scripted pNFS Layouts2012
- Parallel I/O and the metadata wall2011
- Data warehousing and analytics infrastructure at facebook2010
- GFSCommunications of the ACM, 2010
- HBA: Distributed Metadata Management for Large Cluster-Based Storage SystemsIEEE Transactions on Parallel and Distributed Systems, 2008
- Exporting Storage Systems in a Scalable Manner with pNFS2005
- Dynamic Metadata Management for Petabyte-Scale File Systems2005
- The Google file system2003
- Efficient metadata management in large distributed storage systems2003
- Locality-aware request distribution in cluster-based network servers1998