Jerome Soumagne
Research Subjects
Jerome Soumagne has a total of 79 co-authors. Here are the top co-authors by number of shared publications:
Quincey Koziol
Lawrence Berkeley National Laboratory Berkeley California
5 shared publicationsHoujun Tang
Lawrence Berkeley National Laboratory Berkeley California
4 shared publications- John Biddiscombe
Swiss National Supercomputing Centre
4 shared publications - Philip Carns
Argonne National Laboratory (ANL), Lemont, USA
4 shared publications Richard Warren
The HDF Group Champaign Illinois
4 shared publicationsSuren Byna
Lawrence Berkeley National Laboratory Berkeley California
4 shared publications
Latest Publications
- 30 March 2026
- Published by Agence Bibliographique de l'Enseignement Supérieur
An in-situ visualization approach for parallel coupling and steering of simulations through distributed shared memory files
Une approche de visualisation in-situ pour le couplage parallèle et le pilotage de simulations à travers des fichiers en mémoire distribuée partagée: Les codes de simulation devenant plus performants et plus interactifs, il est important de suivre l'avancement d'une simulation in-situ, en réalisant non seulement la visualisation mais aussi l'analyse des données en même temps qu'elles sont générées. Suivre l'avancement ou réaliser le post-traitement des données de simulation in-situ présente un avantage évident par rapport à l'approche conventionnelle consistant à sauvegarder—et à recharger—à partir d'un système de fichiers; le temps et l'espace pris pour écrire et ensuite lire les données à partir du disque est un goulet d'étranglement significatif pour la simulation et les étapes consécutives de post-traitement. Par ailleurs, la simulation peut être arrêtée, modifiée, ou potentiellement pilotée, conservant ainsi les ressources CPU.Nous présentons dans cette thèse une approche de couplage faible qui permet à une simulation de transférer des données vers un serveur de visualisation via l'utilisation de fichiers en mémoire. Nous montrons dans cette étude comment l'interface, implémentée au-dessus d'un format hiérarchique de données (HDF5), nous permet de réduire efficacement le goulet d'étranglement introduit par les I/Os en utilisant des stratégies efficaces de communication et de configuration des données. Pour le pilotage, nous présentons une interface qui permet non seulement la modification de simples paramètres, mais également le remaillage complet de grilles ou des opérations impliquant la régénérationde grandeurs numériques sur le domaine entier de calcul d'être effectués. Cette approche, testée et validée sur deux cas-tests industriels, est suffisamment générique pour qu'aucune connaissance particulière du modèle de données sous-jacent ne soit requise.
- 30 March 2026
- Published by Agence Bibliographique de l'Enseignement Supérieur
An in-situ visualization approach for parallel coupling and steering of simulations through distributed shared memory files
Une approche de visualisation in-situ pour le couplage parallèle et le pilotage de simulations à travers des fichiers en mémoire distribuée partagée: Les codes de simulation devenant plus performants et plus interactifs, il est important de suivre l'avancement d'une simulation in-situ, en réalisant non seulement la visualisation mais aussi l'analyse des données en même temps qu'elles sont générées. Suivre l'avancement ou réaliser le post-traitement des données de simulation in-situ présente un avantage évident par rapport à l'approche conventionnelle consistant à sauvegarder—et à recharger—à partir d'un système de fichiers; le temps et l'espace pris pour écrire et ensuite lire les données à partir du disque est un goulet d'étranglement significatif pour la simulation et les étapes consécutives de post-traitement. Par ailleurs, la simulation peut être arrêtée, modifiée, ou potentiellement pilotée, conservant ainsi les ressources CPU.Nous présentons dans cette thèse une approche de couplage faible qui permet à une simulation de transférer des données vers un serveur de visualisation via l'utilisation de fichiers en mémoire. Nous montrons dans cette étude comment l'interface, implémentée au-dessus d'un format hiérarchique de données (HDF5), nous permet de réduire efficacement le goulet d'étranglement introduit par les I/Os en utilisant des stratégies efficaces de communication et de configuration des données. Pour le pilotage, nous présentons une interface qui permet non seulement la modification de simples paramètres, mais également le remaillage complet de grilles ou des opérations impliquant la régénérationde grandeurs numériques sur le domaine entier de calcul d'être effectués. Cette approche, testée et validée sur deux cas-tests industriels, est suffisamment générique pour qu'aucune connaissance particulière du modèle de données sous-jacent ne soit requise.
- Open Access
- 4 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 7-18
Enhancing RPC on Slingshot for Aurora’s DAOS Storage System
- 4 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 7-18
Open AccessEnhancing RPC on Slingshot for Aurora’s DAOS Storage System
- Open Access
- 4 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 79-92
Towards Empirical Roofline Modeling of Distributed Data Services: Mapping the Boundaries of RPC Throughput
- 4 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 79-92
Open AccessTowards Empirical Roofline Modeling of Distributed Data Services: Mapping the Boundaries of RPC Throughput
- 14 December 2024
- Published by Springer Nature
- p. 185-195
High Performance Fabric Support in DAOS
- 1 July 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Computing in Science & Engineering
- Vol. 25 (4) , 35-41
Mochi: A Case Study in Translational Computer Science for High-Performance Computing Data Management
High-performance computing (HPC) has become an indispensable tool for solving diverse problems in science and engineering. Harnessing the power of HPC is not just a matter of efficient computation, however; it also calls for the efficient management of vast quantities of scientific data. This presents daunting challenges: rapidly evolving storage technology has motivated a shift toward increasingly complex, heterogeneous storage architectures that are difficult to optimize, and scientific data management needs have become every bit as diverse as the application domains that drive them. There is a clear need for agile, adaptable storage solutions that can be customized for the task and platform at hand. This motivated the establishment of the Mochi composable data service project. The Mochi project provides a library of robust, reusable, modular, and connectable data management components and microservices along with a methodology for composing them into specialized distributed data services. Mochi enables rapid deployment of custom data services with a high degree of developer productivity while still effectively leveraging cutting-edge HPC hardware. This article explores how the principles of translational computer science have been applied in practice in Mochi to achieve these goals.
- 1 July 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Computing in Science & Engineering
- Vol. 25 (4) , 35-41
Mochi: A Case Study in Translational Computer Science for High-Performance Computing Data Management
High-performance computing (HPC) has become an indispensable tool for solving diverse problems in science and engineering. Harnessing the power of HPC is not just a matter of efficient computation, however; it also calls for the efficient management of vast quantities of scientific data. This presents daunting challenges: rapidly evolving storage technology has motivated a shift toward increasingly complex, heterogeneous storage architectures that are difficult to optimize, and scientific data management needs have become every bit as diverse as the application domains that drive them. There is a clear need for agile, adaptable storage solutions that can be customized for the task and platform at hand. This motivated the establishment of the Mochi composable data service project. The Mochi project provides a library of robust, reusable, modular, and connectable data management components and microservices along with a methodology for composing them into specialized distributed data services. Mochi enables rapid deployment of custom data services with a high degree of developer productivity while still effectively leveraging cutting-edge HPC hardware. This article explores how the principles of translational computer science have been applied in practice in Mochi to achieve these goals.
- 19 July 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 903-914
Accelerating HDF5 I/O for Exascale Using DAOS
The Hierarchical Data Format 5 (HDF5) has long been defined as one of the most prominent data models, binary file formats and I/O libraries for storing and managing scientific data. Introduced in the late 90s when POSIX I/O was the standard, the library has since then been continuously improved to respond and adapt to the ever-growing demands of high-performance computing (HPC) software and hardware. Given the limitations of POSIX I/O and with the emergence of new technologies such as object stores, non-volatile memory, and SSDs, the need for an interface that can efficiently store and access data at scale through new paradigms has become more and more pressing. The Distributed Asynchronous Object Storage (DAOS) file system is an emerging file system that aims at responding to those demands by taking disk-based storage out of the loop. We present in this article the research efforts that have been taking place to prepare the HDF5 library for Exascale using DAOS. By enabling and defining a new storage file format, we focus on the benefits that it delivers to the applications in terms of features and performance.
- 19 July 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (4) , 903-914
Accelerating HDF5 I/O for Exascale Using DAOS
The Hierarchical Data Format 5 (HDF5) has long been defined as one of the most prominent data models, binary file formats and I/O libraries for storing and managing scientific data. Introduced in the late 90s when POSIX I/O was the standard, the library has since then been continuously improved to respond and adapt to the ever-growing demands of high-performance computing (HPC) software and hardware. Given the limitations of POSIX I/O and with the emergence of new technologies such as object stores, non-volatile memory, and SSDs, the need for an interface that can efficiently store and access data at scale through new paradigms has become more and more pressing. The Distributed Asynchronous Object Storage (DAOS) file system is an emerging file system that aims at responding to those demands by taking disk-based storage out of the loop. We present in this article the research efforts that have been taking place to prepare the HDF5 library for Exascale using DAOS. By enabling and defining a new storage file format, we focus on the benefits that it delivers to the applications in terms of features and performance.
- 9 March 2020
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 32 (20) , e5715
Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage
Summary: Object storage technologies that take advantage of multitier storage on HPC systems are emerging. However, to use these technologies at present, applications have to be modified significantly from current I/O libraries. HDF5, a widely used I/O middleware on HPC systems, provides a virtual object layer (VOL) that allows applications to connect to different storage mechanisms transparently without requiring significant code modifications. We recently designed the proactive data containers (PDC) object‐centric storage system that provides the capabilities of transparent, asynchronous, and autonomous data movement taking advantage of multiple storage tiers—a decision that has so far been left upon the user on most current systems. To enable PDC's features through HDF5 without modifying application codes, we have developed an HDF5 VOL connector that interfaces with PDC. We present in this article the connector interface and evaluate its performance on Cori, a Cray XC40 supercomputer located at the National Energy Research Scientific Computing Center (NERSC). Our evaluation demonstrates up to an 8× improvement compared with HDF5 that has the most recent optimizations.
- 9 March 2020
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 32 (20) , e5715
Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage
Summary: Object storage technologies that take advantage of multitier storage on HPC systems are emerging. However, to use these technologies at present, applications have to be modified significantly from current I/O libraries. HDF5, a widely used I/O middleware on HPC systems, provides a virtual object layer (VOL) that allows applications to connect to different storage mechanisms transparently without requiring significant code modifications. We recently designed the proactive data containers (PDC) object‐centric storage system that provides the capabilities of transparent, asynchronous, and autonomous data movement taking advantage of multiple storage tiers—a decision that has so far been left upon the user on most current systems. To enable PDC's features through HDF5 without modifying application codes, we have developed an HDF5 VOL connector that interfaces with PDC. We present in this article the connector interface and evaluate its performance on Cori, a Cray XC40 supercomputer located at the National Energy Research Scientific Computing Center (NERSC). Our evaluation demonstrates up to an 8× improvement compared with HDF5 that has the most recent optimizations.
- 17 January 2020
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 35 (1) , 145-160
ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems
- 17 January 2020
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 35 (1) , 145-160
ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems
- 17 January 2020
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 35 (1) , 121-144
Mochi: Composing Data Services for High-Performance Computing Environments
- 17 January 2020
- Published by Springer Nature in Journal of Computer Science and Technology
- Vol. 35 (1) , 121-144
Mochi: Composing Data Services for High-Performance Computing Environments
- 1 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 73-82
Analysis in the Data Path of an Object-Centric Data Management System
Emerging high performance computing (HPC) systems are expected to be deployed with an unprecedented level of complexity due to a deep system memory and storage hierarchy. Efficient and scalable methods of data management and movement through the multi-level storage hierarchy of upcoming HPC systems will be critical for scientific applications at exascale. In this paper, we propose in locus analysis that allows registering user-defined functions (UDFs) and running those functions automatically while the data is moving between levels of a storage hierarchy. We implement this analysis in the data path approach in our object-centric data management system, called Proactive Data Containers (PDC). The transparent invocation of analysis functions as part of PDC object mapping is an optimized approach to minimize latency to access data as it moves within the storage hierarchy. Because a user defined analysis or transform function will be invoked automatically by the PDC runtime, the user simply registers their functions for PDC to identify the function name as well as the required list of actual parameters. To demonstrate the validity and flexibility of this analysis approach, we have implemented several scientific analysis kernels to compare against other HPC analysis-oriented approaches.
- 1 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 73-82
Analysis in the Data Path of an Object-Centric Data Management System
Emerging high performance computing (HPC) systems are expected to be deployed with an unprecedented level of complexity due to a deep system memory and storage hierarchy. Efficient and scalable methods of data management and movement through the multi-level storage hierarchy of upcoming HPC systems will be critical for scientific applications at exascale. In this paper, we propose in locus analysis that allows registering user-defined functions (UDFs) and running those functions automatically while the data is moving between levels of a storage hierarchy. We implement this analysis in the data path approach in our object-centric data management system, called Proactive Data Containers (PDC). The transparent invocation of analysis functions as part of PDC object mapping is an optimized approach to minimize latency to access data as it moves within the storage hierarchy. Because a user defined analysis or transform function will be invoked automatically by the PDC runtime, the user simply registers their functions for PDC to identify the function name as well as the required list of actual parameters. To demonstrate the validity and flexibility of this analysis approach, we have implemented several scientific analysis kernels to compare against other HPC analysis-oriented approaches.