Metall: A Persistent Memory Allocator For Data-Centric Analytics
Preprint
Abstract
Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion step is often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall's high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. This study's outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- UMap: Enabling Application-driven Optimizations for Page Management2019
- Skyway2018
- Mathematical foundations of the GraphBLAS2016
- Towards a Distributed Large-Scale Dynamic Graph Data Store2016
- Exascale computing and big dataCommunications of the ACM, 2015
- SuperMalloc: a super fast multithreaded malloc for 64-bit machines2015
- Variable-Order de Bruijn Graphs2015
- DI-MMAP: A High Performance Memory-Map Runtime for Data-Intensive Applications2012
- NVMalloc: Exposing an Aggregate SSD Store as a Memory Partition in Extreme-Scale Machines2012
- Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memory2010