Efficient Provenance Management via Clustering and Hybrid Storage in Big Data Environments
- 25 March 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Big Data
- Vol. 6 (4) , 792-803
Abstract
Provenance is a type of metadata that records the creation and transformation of data objects. It has been applied to a wide variety of areas such as security, search, and experimental documentation. However, provenance usually has a vast amount of data with its rapid growth rate which hinders the effective extraction and application of provenance. This paper proposes an efficient provenance management system via clustering and hybrid storage. Specifically, we propose a Provenance-Based Label Propagation Algorithm which is able to regularize and cluster a large number of irregular provenance. Then, we use separate physical storage mediums, such as SSD and HDD, to store hot and cold data separately, and implement a hot/cold scheduling scheme which can update and schedule data between them automatically. Besides, we implement a feedback mechanism which can locate and compress the rarely used cold data according to the query request. The experimental test shows that the system can significantly improve provenance query performance with a small run-time overhead.Keywords
Funding Information
- National Natural Science Foundation of China (U1705261, 61402189, 61821003, 61772222)
- CCF-NSFOCUS Kunpeng Research Fund
- Wuhan Application Basic Research Program (2017010201010104)
- Hubei Natural Science and Technology Foundation (2017CFB304)
This publication has 24 references indexed in Scilit:
- Stochastic Modeling and Performance Analysis of Migration-Enabled and Error-Prone CloudsIEEE Transactions on Industrial Informatics, 2015
- HDStore: An SSD/HDD Hybrid Distributed Storage Scheme for Large-Scale Data2014
- Evaluation of a Hybrid Approach for Efficient Provenance StorageACM Transactions on Storage, 2013
- Local clustering in provenance graphs2013
- A hybrid approach for efficient provenance storage2012
- Efficient provenance storage for relational queries2012
- SPADE: Support for Provenance Auditing in Distributed Environments2012
- Causality-based versioningACM Transactions on Storage, 2009
- Near linear time algorithm to detect community structures in large-scale networksPhysical Review E, 2007
- Michigan Molecular Interactions (MiMI): putting the jigsaw puzzle togetherNucleic Acids Research, 2006