Combining HW/SW Mechanisms to Improve NUMA Performance of Multi-GPU Systems
Authors:
- Vinson YoungVYVinson Young
- Aamer JaleelAJAamer Jaleel
- Evgeny BolotinEBEvgeny Bolotin
- Eiman EbrahimiEEEiman Ebrahimi
- David NellansDNDavid Nellans
- Oreste VillaOVOreste Villa
- 1 October 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 339-351
Abstract
Historically, improvement in GPU performance has been tightly coupled with transistor scaling. As Moore's Law slows down, performance of single GPUs may ultimately plateau. To continue GPU performance scaling, multiple GPUs can be connected using system-level interconnects. However, limited inter-GPU interconnect bandwidth (e.g., 64GB/s) can hurt multi-GPU performance when there are frequent remote GPU memory accesses. Traditional GPUs rely on page migration to service the memory accesses from local memory instead. Page migration fails when the page is simultaneously shared between multiple GPUs in the system. As such, recent proposals enhance the software runtime system to replicate read-only shared pages in local memory. Unfortunately, such practice fails when there are frequent remote memory accesses to read-write shared pages. To address this problem, recent proposals cache remote shared data in the GPU last-level-cache (LLC). Unfortunately, remote data caching also fails when the shared-data working-set exceeds the available GPU LLC size. This paper conducts a combined performance analysis of state-of-the-art software and hardware mechanisms to improve NUMA performance of multi-GPU systems. Our evaluations on a 4-node multi-GPU system reveal that the combination of work scheduling, page placement, page migration, page replication, and caching remote data still incurs a 47% slowdown relative to an ideal NUMA-GPU system. This is because the shared memory footprint tends to be significantly larger than the GPU LLC size and can not be replicated by software because the shared footprint has read-write property. Thus, we show that existing NUMA-aware software solutions require hardware support to address the NUMA bandwidth bottleneck. We propose Caching Remote Data in Video Memory (CARVE), a hardware mechanism that stores recently accessed remote shared data in a dedicated region of the GPU memory. CARVE outperforms state-of-the-art NUMA mechanisms and is within 6% the performance of an ideal NUMA-GPU system. A design space analysis on supporting cache coherence is also investigated. Overall, we show that dedicating only 3% of GPU memory eliminates NUMA bandwidth bottlenecks while incurring negligible performance overheads due to the reduced GPU memory capacity.Keywords
This publication has 30 references indexed in Scilit:
- Memory access patterns2015
- Automatic Parallelization of Kernels in Shared-Memory Multi-GPU Nodes2015
- QuickRelease: A throughput-oriented approach to release consistency on GPUs2014
- Die-stacked DRAM caches for servers2013
- Traffic management2013
- Cache coherence for GPU architectures2013
- A scalable 0.128-to-1Tb/s 0.8-to-2.6pJ/b 64-lane parallel I/O in 32nm CMOS2013
- A Mostly-Clean DRAM Cache for Effective Hit Speculation and Self-Balancing Dispatch2012
- An argument for simple COMA2002
- Design and analysis of a scalable cache coherence scheme based on clocks and timestampsIEEE Transactions on Parallel and Distributed Systems, 1992