Union: An Automatic Workload Manager for Accelerating Network Simulation
Preprint
Abstract
With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time.Keywords
All Related Versions
- Version 1, 2024-03-25, ArXiv
- Version 2, 2024-04-03, ArXiv
- Published version: , 821.
This publication has 5 references indexed in Scilit:
- Mitigating network noise on Dragonfly networks through application-aware routing2019
- Argobots: A Lightweight Low-Level Threading and Tasking FrameworkIEEE Transactions on Parallel and Distributed Systems, 2017
- Enabling Parallel Simulation of Large-Scale HPC Network SystemsIEEE Transactions on Parallel and Distributed Systems, 2016
- The Design and Implementation of a Domain-Specific Language for Network Performance TestingIEEE Transactions on Parallel and Distributed Systems, 2007
- ROSS: A high-performance, low-memory, modular Time Warp systemJournal of Parallel and Distributed Computing, 2002