Cache Cohort GPU Scheduling

Authors:
Open Access
Publisher website
Google Scholar
Abstract
With the ever-improving computation capability of GPUs, there is an increasing demand for higher memory bandwidth to supply the GPU cores with data. One way to improve effective memory bandwidth is with larger caches, and we have seen GPU vendors recently invest in very large last-level caches (LLCs). The key challenge is to maximize cache utilization and keep the active working set on-chip during the lifetime of a kernel. Software tiling optimizations can help, but programmers alone cannot anticipate dynamic cache capacity contention. To this end, we introduce the concept of cache cohorts: groups of kernels that share the same working set and thus can be efficiently scheduled together. Our approach uses the GPU command processor (CP) to track the occupancy of the LLC and throttles scheduling of the next cohort if there is no room. To evaluate the design space, we modify the existing GPU firmware on the AMD RX 7600 and RX 7900 XTX GPUs to prototype the scheduling scheme using real hardware. Our preliminary results show up to 27% reduction in execution time using the cache optimized scheduling technique on RX 7900 XTX.

This publication has 24 references indexed in Scilit: