Abstract
Artificial Intelligence (AI) and Machine Learning (ML) workflows are increasingly prevalent in High Performance Computing (HPC) centers, requiring additional support from campus research computing centers like Georgia Institute of Technology (GT) Partnership for an Advanced Computing Environment (PACE). AI datasets usually contain many small files and a large total size; it wastes storage space if every researcher downloads copies. Hence a shared dataset is desired. The challenge is how to share the AI dataset to support multiple concurrent AI workflows efficiently. HPC filesystem choices, low-level filesystem configuration, and AI pipeline data loader selection play essential roles. This paper demonstrates a systematic performance test to guide us in implementing a shared AI data service at PACE.