Union: An Automatic Workload Manager for Accelerating Network Simulation
Authors:
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 821-830
Abstract
With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Watch Out for the Bully! Job Interference Study on Dragonfly Network2016
- Evaluating HPC Networks via Simulation of Parallel Workloads2016
- Modular HPC I/O Characterization with Darshan2016
- Enabling Parallel Simulation of Large-Scale HPC Network SystemsIEEE Transactions on Parallel and Distributed Systems, 2016
- Quiet Neighborhoods: Key to Protect Job Performance Predictability2015
- A detailed and flexible cycle-accurate Network-on-Chip simulator2013
- Technology-Driven, Highly-Scalable Dragonfly TopologyACM SIGARCH Computer Architecture News, 2008
- PinACM SIGPLAN Notices, 2005
- coNCePTual: a network correctness and performance testing language2004
- ROSS: A high-performance, low-memory, modular Time Warp systemJournal of Parallel and Distributed Computing, 2002