Oasis : Controlling Data Migration in Expansion of Object-based Storage Systems
Open Access
- 19 January 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Storage
- Vol. 19 (1) , 1-22
Abstract
Object-based storage systems have been widely used for various scenarios such as file storage, block storage, blob (e.g., large videos) storage, and so on, where the data is placed among a large number of object storage devices (OSDs). Data placement is critical for the scalability of decentralized object-based storage systems. The state-of-the-art CRUSH placement method is a decentralized algorithm that deterministically places object replicas onto storage devices without relying on a central directory. While enjoying the benefits of decentralization such as high scalability, robustness, and performance, CRUSH-based storage systems suffer from uncontrolled data migration when expanding the capacity of the storage clusters (i.e., adding new OSDs), which is determined by the nature of CRUSH and will cause significant performance degradation when the expansion is nontrivial. This article presents MapX , a novel extension to CRUSH that uses an extra time-dimension mapping (from object creation times to cluster expansion times) for controlling data migration after cluster expansions. Each expansion is viewed as a new layer of the CRUSH map represented by a virtual node beneath the CRUSH root. MapX controls the mapping from objects onto layers by manipulating the timestamps of the intermediate placement groups (PGs). MapX is applicable to a large variety of object-based storage scenarios where object timestamps can be maintained as higher-level metadata. We have applied MapX to the state-of-the-art Ceph-RBD (RADOS Block Device) to implement a migration-controllable, decentralized object-based block store (called Oasis ). Oasis extends the RBD metadata structure to maintain and retrieve approximate object creation times (for migration control) at the granularity of expansion layers. Experimental results show that the MapX -based Oasis block store outperforms the CRUSH-based Ceph-RBD (which is busy in migrating objects after expansions) by 3.17× ∼ 4.31× in tail latency, and 76.3% (respectively, 83.8%) in IOPS for reads (respectively, writes).Keywords
Funding Information
- National Key R&D Program of China (2022YFB4500302)
- National Natural Science Foundation of China (61872376, 61932001, and 62072306)
- Program of Shanghai Academic Research Leader (20XD1402100)
This publication has 34 references indexed in Scilit:
- CassandraACM SIGOPS Operating Systems Review, 2010
- BigtableACM Transactions on Computer Systems, 2008
- MapReduceCommunications of the ACM, 2008
- DynamoACM SIGOPS Operating Systems Review, 2007
- Tapestry: A Resilient Global-Scale Overlay for Service DeploymentIEEE Journal on Selected Areas in Communications, 2004
- ChordACM SIGCOMM Computer Communication Review, 2001
- The Zebra striped network file systemACM Transactions on Computer Systems, 1995
- RAID: high-performance, reliable secondary storageACM Computing Surveys, 1994
- Linearizability: a correctness condition for concurrent objectsACM Transactions on Programming Languages and Systems, 1990
- A fast file system for UNIXACM Transactions on Computer Systems, 1984