Javier Garcia Blas

As data-heavy applications continue to become more popular, High-Performance Computing (HPC) storage systems are under immense strain in terms of both capacity and speed. Although modern innovations like NVMe and persistent memory provide substantial performance enhancements, equipping entire systems exclusively with high-speed storage is impractical in terms of cost, making multi-tier storage architectures essential. Compounding this challenge, modern workloads across HPC, data analytics, and artificial intelligence exhibit increasingly heterogeneous and dynamic I/O access patterns. While traditional storage was designed for predictable, batch-oriented tasks, today’s applications generate bursty, mixed traffic that strains the rigidity of existing storage hierarchies. Consequently, there is a need to shift toward adaptive, malleable designs that bridge the gap between evolving application behaviors and underlying infrastructures.To address these demands, ad-hoc file systems have emerged as a practical solution by pooling local compute-node resources, such as RAM and local persistent storage, to create highly adaptable, temporary file systems that are customized to the specific behavior of HPC workloads. This work explores the malleability capabilities of HERCULES, a distributed ad-hoc in-memory storage system, detailing its malleability model design. The system incorporates runtime malleability and fault-tolerance mechanisms, enabling dynamic data migration through flexible allocation policies to maximize resource utilization. Comprehensive evaluation results show that HERCULES outperforms Lustre and BeeGFS, two widely used parallel file systems in high-performance computing environments.