An Integrated Indexing and Search Service for Distributed File Systems
- 27 April 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 31 (10) , 2375-2391
Abstract
Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this article, we present TagIt, a scalable data management service framework aimed at scientific datasets, which can be integrated into prevalent distributed file system architectures. A key feature of TagIt is a scalable, distributed metadata indexing framework, which facilitates a flexible tagging capability to support data discovery. Furthermore, the tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. We have integrated TagIt into two popular distributed file systems, i.e., GlusterFS and CephFS. Our evaluation demonstrates that TagIt can expedite data search operation by up to 10× over the extant decoupled approach.Keywords
Funding Information
- U.S. DOE's Scientific data management program
- National Science Foundation (CNS-1615411, CNS-1405697, CNS-1565314)
- National Research Foundation of Korea (2018R1A1A1A05079398)
- Oak Ridge Leadership Computing Facility (DE-AC05-00OR22725)
This publication has 27 references indexed in Scilit:
- Comparative I/O workload characterization of two leadership class storage clusters2015
- Mantle2015
- A scalable search engine for mass storage smart objectsProceedings of the VLDB Endowment, 2015
- Propeller: A Scalable Real-Time File-Search Service in Distributed Systems2014
- LazyBase2012
- RADOS2007
- CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data2006
- Scalable security for large, high performance storage systems2006
- Semantic file systems1991
- A fast file system for UNIXACM Transactions on Computer Systems, 1984