Exploring Research Dataset-Sharing Strategies for Concurrent AI Workflows
Authors:
Open Access
- 17 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-4
Abstract
Artificial Intelligence (AI) and Machine Learning (ML) workflows are increasingly prevalent in High Performance Computing (HPC) centers, requiring additional support from campus research computing centers like Georgia Institute of Technology (GT) Partnership for an Advanced Computing Environment (PACE). AI datasets usually contain many small files and a large total size; it wastes storage space if every researcher downloads copies. Hence a shared dataset is desired. The challenge is how to share the AI dataset to support multiple concurrent AI workflows efficiently. HPC filesystem choices, low-level filesystem configuration, and AI pipeline data loader selection play essential roles. This paper demonstrates a systematic performance test to guide us in implementing a shared AI data service at PACE.Keywords
This publication has 6 references indexed in Scilit:
- ICE 2.0: Restructuring and Growing an Instructional HPC Cluster2023
- Protein complex prediction with AlphaFold-MultimerbioRxiv, 2021
- Highly accurate protein structure prediction with AlphaFoldNature, 2021
- High Performance I/O For Large Scale Deep Learning2019
- Aggregated Residual Transformations for Deep Neural Networks2017
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015