MoCA: Memory-Centric, Adaptive Execution for Multi-Tenant Deep Neural Networks
Authors:
- 1 February 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 828-841
Abstract
Driven by the wide adoption of deep neural networks (DNNs) across different application domains, multi-tenancy execution, where multiple DNNs are deployed simultaneously on the same hardware, has been proposed to satisfy the latency requirements of different applications while improving the overall system utilization. However, multi-tenancy execution could lead to undesired system-level resource contention, causing quality-of-service (QoS) degradation for latency-critical applications.To address this challenge, we propose MoCA 1 , an adaptive multi-tenancy system for DNN accelerators. Unlike existing solutions that focus on compute resource partition, MoCA dynamically manages shared memory resources of co-located applications to meet their QoS targets. Specifically, MoCA leverages the regularities in both DNN operators and accelerators to dynamically modulate memory access rates based on their latency targets and user-defined priorities so that co-located applications get the resources they demand without significantly starving their co-runners. We demonstrate that MoCA improves the satisfaction rate of the service level agreement (SLA) up to 3.9× (1.8× average), system throughput by 2.3× (1.7× average), and fairness by 1.3× (1.2× average), compared to prior work.Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Baymax2016
- Heracles2015
- Going deeper with convolutions2015
- Quasar2014
- A hardware evaluation of cache partitioning to improve utilization and energy-efficiency while preserving responsivenessACM SIGARCH Computer Architecture News, 2013
- ReQoS2013
- Chisel2012
- Fairness via Source ThrottlingACM Transactions on Computer Systems, 2012
- CoQoS: Coordinating QoS-aware shared resources in NoC-based SoCsJournal of Parallel and Distributed Computing, 2011
- Effective Management of DRAM Bandwidth in Multicore Processors16th International Conference on Parallel Architecture and Compilation Techniques (PACT 2007), 2007