Daniel Araújo De Medeiros
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Daniel Araújo De Medeiros has a total of 49 co-authors. Here are the top co-authors by number of shared publications:
- Stefano Markidis
KTH Royal Institute of Technology, Stockholm, Sweden
2 shared publications - André Brinkmann
Saarland University, Saarbrücken, Germany
1 shared publication - Buket Benek Gursoy
Irish Centre for High-End Computing, Ireland
1 shared publication - Damien Chapon
Université Paris-Saclay
1 shared publication - Daniel Caviedes-Voullième
Chair for Environmental Fluid Dynamics and Modeling, TUD Dresden University of Technology, Dresden, Germany
1 shared publication - Eric B. Gregory
Jülich Supercomputer Centre, Jülich, Germany
1 shared publication
Latest Publications
- Open Access
- 7 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 94-100
IO-SEA: Storage I/O and Data Management for Exascale Architectures
The new emerging scientific workloads to be executed in the upcoming exascale supercomputers face major challenges in terms of storage, given their extreme volume of data. In particular, intelligent data placement, instrumentation, and workflow handling are central to application performance. The IO-SEA project developed multiple solutions to aid the scientific community in adressing these challenges: a Workflow Manager, a hierarchical storage management system, and a semantic API for storage. All of these major products incorporate additional minor products that support their mission. In this paper, we discuss both the roles of all these products and how they can assist the scientific community in achieving exascale performance.
- 7 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 94-100
Open AccessIO-SEA: Storage I/O and Data Management for Exascale Architectures
The new emerging scientific workloads to be executed in the upcoming exascale supercomputers face major challenges in terms of storage, given their extreme volume of data. In particular, intelligent data placement, instrumentation, and workflow handling are central to application performance. The IO-SEA project developed multiple solutions to aid the scientific community in adressing these challenges: a Workflow Manager, a hierarchical storage management system, and a semantic API for storage. All of these major products incorporate additional minor products that support their mission. In this paper, we discuss both the roles of all these products and how they can assist the scientific community in achieving exascale performance.
- Open Access
- 27 February 2023
- Published by Association for Computing Machinery (ACM)
- p. 106-116
LibCOS: Enabling Converged HPC and Cloud Data Stores with MPI
Recently, federated HPC and cloud resources are becoming increasingly strategic for providing diversified and geographically available computing resources. However, accessing data stores across HPC and cloud storage systems is challenging. Many cloud providers use object storage systems to support their clients in storing and retrieving data over the internet. One popular method is REST APIs atop the HTTP protocol, with Amazon’s S3 APIs being supported by most vendors. In contrast, HPC systems are contained within their networks and tend to use parallel file systems with POSIX-like interfaces. This work addresses the challenge of diverse data stores on HPC and cloud systems by providing native object storage support through the unified MPI I/O interface in HPC applications. In particular, we provide a prototype library called LibCOS that transparently enables MPI applications running on HPC systems to access object storage on remote cloud systems. We evaluated LibCOS on a Ceph object storage system and a traditional HPC system. In addition, we conducted performance characterization of core S3 operations that enable individual and collective MPI I/O. Our evaluation in HACC, IOR, and BigSort shows that enabling diverse data stores on HPC and Cloud storage is feasible and can be transparently achieved through the widely adopted MPI I/O. Also, we show that a native object storage system like Ceph could improve the scalability of I/O operations in parallel applications.
- 27 February 2023
- Published by Association for Computing Machinery (ACM)
- p. 106-116
Open AccessLibCOS: Enabling Converged HPC and Cloud Data Stores with MPI
Recently, federated HPC and cloud resources are becoming increasingly strategic for providing diversified and geographically available computing resources. However, accessing data stores across HPC and cloud storage systems is challenging. Many cloud providers use object storage systems to support their clients in storing and retrieving data over the internet. One popular method is REST APIs atop the HTTP protocol, with Amazon’s S3 APIs being supported by most vendors. In contrast, HPC systems are contained within their networks and tend to use parallel file systems with POSIX-like interfaces. This work addresses the challenge of diverse data stores on HPC and cloud systems by providing native object storage support through the unified MPI I/O interface in HPC applications. In particular, we provide a prototype library called LibCOS that transparently enables MPI applications running on HPC systems to access object storage on remote cloud systems. We evaluated LibCOS on a Ceph object storage system and a traditional HPC system. In addition, we conducted performance characterization of core S3 operations that enable individual and collective MPI I/O. Our evaluation in HACC, IOR, and BigSort shows that enabling diverse data stores on HPC and Cloud storage is feasible and can be transparently achieved through the widely adopted MPI I/O. Also, we show that a native object storage system like Ceph could improve the scalability of I/O operations in parallel applications.