Karim Youssef
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Karim Youssef has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Maya Gokhale
Lawrence Livermore National Laboratory
3 shared publications Roger Pearce
Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, Livermore, CA, USA
3 shared publications- Hariharan Devarajan
Lawrence Livermore National Laboratory, Livermore, CA, USA
2 shared publications - Ivy Peng
KTH Royal Institute of Technology
2 shared publications Keita Iwabuchi
Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, Livermore, CA, USA
2 shared publications- Keita Iwabuchi
Lawrence Livermore National Laboratory, 7000 East Avenue, Livermore, CA 94550, USA
2 shared publications
Latest Publications
- Open Access
- 11 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-12
WADO: A Distributed WORM Storage Service for Asynchronous Data Operations
AI-driven scientific workloads increasingly depend on data-intensive input pipelines, where deep learning frameworks must ingest and transform large datasets from hierarchical HPC storage. Existing system-centric data services improve movement and locality between the parallel file system (PFS), node-local storage, and memory. However, they do not directly optimize how input pipeline operations execute across scopes, stage overlap, and resource-specific parallelism. As scale grows, this gap causes worker stalls, contention, and poor hardware utilization. We present WADO, a distributed write-once-read-many (WORM) object-store runtime for data-centric workloads that closes this gap through three coordinated mechanisms: scope-centric processing, explicit pipeline decomposition, and interference-aware explicit parallelism. WADO dynamically maps operations to execution scopes, overlaps stages such as I/O, communication, and transformations, and applies contention-aware concurrency control to match hardware behavior at runtime. Our evaluation shows three main findings: (1) scope-centric processing preserves throughput under scale, improving mixed-operation throughput by up to 1.65 × ; (2) explicit pipeline decomposition converts serialized wait into overlapped progress, delivering up to 2.16 × higher sustained bandwidth; and (3) interference-aware explicit parallelism improves effective bandwidth by up to 4.4 × by avoiding oversubscription collapse. On Unet3D model training, these mechanisms translate to end-to-end gains, improving data loading performance by 4.1 × compared to baseline PyTorch on Lustre, and 1.51 × compared to DYAD, enabled by deeper pipelining, adaptive parallelism, and near-data transformation offloading.
- 11 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessWADO: A Distributed WORM Storage Service for Asynchronous Data Operations
AI-driven scientific workloads increasingly depend on data-intensive input pipelines, where deep learning frameworks must ingest and transform large datasets from hierarchical HPC storage. Existing system-centric data services improve movement and locality between the parallel file system (PFS), node-local storage, and memory. However, they do not directly optimize how input pipeline operations execute across scopes, stage overlap, and resource-specific parallelism. As scale grows, this gap causes worker stalls, contention, and poor hardware utilization. We present WADO, a distributed write-once-read-many (WORM) object-store runtime for data-centric workloads that closes this gap through three coordinated mechanisms: scope-centric processing, explicit pipeline decomposition, and interference-aware explicit parallelism. WADO dynamically maps operations to execution scopes, overlaps stages such as I/O, communication, and transformations, and applies contention-aware concurrency control to match hardware behavior at runtime. Our evaluation shows three main findings: (1) scope-centric processing preserves throughput under scale, improving mixed-operation throughput by up to 1.65 × ; (2) explicit pipeline decomposition converts serialized wait into overlapped progress, delivering up to 2.16 × higher sustained bandwidth; and (3) interference-aware explicit parallelism improves effective bandwidth by up to 4.4 × by avoiding oversubscription collapse. On Unet3D model training, these mechanisms translate to end-to-end gains, improving data loading performance by 4.1 × compared to baseline PyTorch on Lustre, and 1.51 × compared to DYAD, enabled by deeper pipelining, adaptive parallelism, and near-data transformation offloading.
- Open Access
- 13 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 817-826
Metis: Agentic Knowledge Synthesis for Explainable I/O Performance in HPC Systems
I/O performance explainability in HPC requires contextual characterization across the full software and system stack. Contextual characterization identifies the semantics and runtime role of I/O functions. State-of-the-art contextual characterization is still largely manual, but remains highly valuable for explaining bottlenecks and guiding optimization. However, manual contextual characterization is difficult to scale and hard to reproduce as I/O libraries and cross-layer interactions grow in complexity. We present Metis, a framework for systematic characterization of HPC I/O functions that uses agentic LLMs to integrate heterogeneous evidence sources and quantify agents agreement. Across evaluation, Metis improves held-out-category generalization over an MCP Tool baseline (0.90 vs. 0.35), reduces runtime (27.7 s vs. 84.5 s) while increasing throughput (41.8 vs. 14.28 functions/min), and sustains high verifier throughput under federated scaling (330K–1.18M functions/s). These results demonstrate that Metis is an effective and practical approach for explainable characterization of complex HDF5 behavior, enabling more trustworthy and reproducible HPC I/O analysis.
- 13 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 817-826
Open AccessMetis: Agentic Knowledge Synthesis for Explainable I/O Performance in HPC Systems
I/O performance explainability in HPC requires contextual characterization across the full software and system stack. Contextual characterization identifies the semantics and runtime role of I/O functions. State-of-the-art contextual characterization is still largely manual, but remains highly valuable for explaining bottlenecks and guiding optimization. However, manual contextual characterization is difficult to scale and hard to reproduce as I/O libraries and cross-layer interactions grow in complexity. We present Metis, a framework for systematic characterization of HPC I/O functions that uses agentic LLMs to integrate heterogeneous evidence sources and quantify agents agreement. Across evaluation, Metis improves held-out-category generalization over an MCP Tool baseline (0.90 vs. 0.35), reduces runtime (27.7 s vs. 84.5 s) while increasing throughput (41.8 vs. 14.28 functions/min), and sustains high verifier throughput under federated scaling (330K–1.18M functions/s). These results demonstrate that Metis is an effective and practical approach for explainable characterization of complex HDF5 behavior, enabling more trustworthy and reproducible HPC I/O analysis.
- 31 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (2) , 562-574
Optimizing Management of Persistent Data Structures in High-Performance Analytics
Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. We integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.
- 31 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (2) , 562-574
Optimizing Management of Persistent Data Structures in High-Performance Analytics
Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. We integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.
- 20 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Biology and Bioinformatics
- Vol. 22 (4) , 1388-1400
Scalable and Maintainable Distributed Sequence Alignment Using Spark
The exponential growth of genomic data presents a challenge to bioinformatics research. NCBI BLAST, a popular pairwise sequence alignment tool, does not scale with the hundreds of gigabytes (GB) of sequenced data. Therefore, mpiBLAST was widely adopted and scaled up to 65,536 processors. However, mpiBLAST is tightly coupled with an obsolete NCBI BLAST version, creating a challenge to upgrading mpiBLAST with the ever-changing NCBI BLAST code. Recent parallel BLAST implementations, like SparkBLAST, use parallelism wrappers separate from NCBI BLAST to overcome this issue. However, query partitioning, a parallel method that duplicates the genome database on each compute node, makes SparkBLAST scale poorly with databases larger than a single node's memory. Thus, no parallel BLAST utility simultaneously addresses performance, scalability, and software maintainability. To fill this gap, we introduce SparkLeBLAST, a parallel BLAST tool that uses the Spark framework and efficient data partitioning to combine mpiBLAST's performance and scalability with SparkBLAST's simplicity and maintainability. SparkLeBLAST democratizes scalable genomic analysis for domain scientists without extensive distributed computing experience. SparkLeBLAST runs up to 6.68× faster than SparkBLAST. SparkLeBLAST also accelerates taxonomic assignment of COVID-19 genomic diversity analysis by 20.9× as it speeds up the BLAST search component by 88.6× using 128 compute nodes.
- 20 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Biology and Bioinformatics
- Vol. 22 (4) , 1388-1400
Scalable and Maintainable Distributed Sequence Alignment Using Spark
The exponential growth of genomic data presents a challenge to bioinformatics research. NCBI BLAST, a popular pairwise sequence alignment tool, does not scale with the hundreds of gigabytes (GB) of sequenced data. Therefore, mpiBLAST was widely adopted and scaled up to 65,536 processors. However, mpiBLAST is tightly coupled with an obsolete NCBI BLAST version, creating a challenge to upgrading mpiBLAST with the ever-changing NCBI BLAST code. Recent parallel BLAST implementations, like SparkBLAST, use parallelism wrappers separate from NCBI BLAST to overcome this issue. However, query partitioning, a parallel method that duplicates the genome database on each compute node, makes SparkBLAST scale poorly with databases larger than a single node's memory. Thus, no parallel BLAST utility simultaneously addresses performance, scalability, and software maintainability. To fill this gap, we introduce SparkLeBLAST, a parallel BLAST tool that uses the Spark framework and efficient data partitioning to combine mpiBLAST's performance and scalability with SparkBLAST's simplicity and maintainability. SparkLeBLAST democratizes scalable genomic analysis for domain scientists without extensive distributed computing experience. SparkLeBLAST runs up to 6.68× faster than SparkBLAST. SparkLeBLAST also accelerates taxonomic assignment of COVID-19 genomic diversity analysis by 20.9× as it speeds up the BLAST search component by 88.6× using 128 compute nodes.
- 5 December 2024
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 39 (2) , 269-282
UMap: An application-oriented user level memory mapping library
Exploiting the prominent role of complex memories in exascale node architecture, the UMap page fault handler offers new capabilities to access large memory-mapped data sets directly. UMap provides flexible configuration options to customize page handling to each application, including analysis of massive observational and simulation data sets. The high-performance design features I/O decoupling, dynamic load balancing, and application-level controls. Page faults triggered by application threads and processes accessing data mapped to a UMapp’ed region are handled via the Linux userfaultfd protocol, an asynchronous message-oriented kernel-user communication mechanism that avoids the context switch penalty of traditional signal fault handlers. UMap is fully open source. In this paper, we give an overview of the UMap library architecture, its extensible plugin architecture, and the use/performance of UMap in emerging heterogeneous memory hierarchies such as near-node Non-volatile Memory (NVM) and network attached memories. We highlight new capabilities in two pagefault management plugins, the NetworkStore and SparseStore. We demonstrate the integration between UMap and multiple ECP products including Caliper, Metall, ZFP, Mochi, and Ripples.
- 5 December 2024
- Published by SAGE Publications in The International Journal of High Performance Computing Applications
- Vol. 39 (2) , 269-282
UMap: An application-oriented user level memory mapping library
Exploiting the prominent role of complex memories in exascale node architecture, the UMap page fault handler offers new capabilities to access large memory-mapped data sets directly. UMap provides flexible configuration options to customize page handling to each application, including analysis of massive observational and simulation data sets. The high-performance design features I/O decoupling, dynamic load balancing, and application-level controls. Page faults triggered by application threads and processes accessing data mapped to a UMapp’ed region are handled via the Linux userfaultfd protocol, an asynchronous message-oriented kernel-user communication mechanism that avoids the context switch penalty of traditional signal fault handlers. UMap is fully open source. In this paper, we give an overview of the UMap library architecture, its extensible plugin architecture, and the use/performance of UMap in emerging heterogeneous memory hierarchies such as near-node Non-volatile Memory (NVM) and network attached memories. We highlight new capabilities in two pagefault management plugins, the NetworkStore and SparseStore. We demonstrate the integration between UMap and multiple ECP products including Caliper, Metall, ZFP, Mochi, and Ripples.
- 23 March 2022
- Published by Elsevier in Parallel Computing
- Vol. 111
Metall: A persistent memory allocator for data-centric analytics
- 23 March 2022
- Published by Elsevier in Parallel Computing
- Vol. 111
Metall: A persistent memory allocator for data-centric analytics
- 3 June 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 866-877
Enabling Scalable and Extensible Memory-Mapped Datastores in Userspace
Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9${\times}$ . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.- 3 June 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 866-877
Enabling Scalable and Extensible Memory-Mapped Datastores in Userspace
Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9${\times}$ . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.