Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms
- 30 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 45 (4) , 29-36
Abstract
Cloud platforms have deployed hardware accelerators like neural processing units (NPUs) for machine learning (ML) inference services. To maximize resource utilization while ensuring quality of service, a natural approach is to virtualize NPUs for efficient resource sharing for multi-tenant ML services. However, virtualizing NPUs is challenging. This is not only due to the lack of system abstraction support for NPU hardware, but also due to the lack of architectural and ISA support for fine-grained operator scheduling. This article presents Neu10, an NPU virtualization framework consisting of (1) an abstraction called vNPU for fine-grained virtualization of the heterogeneous compute units in a physical NPU (pNPU); (2) a vNPU allocator that enables pay-as-you-go pricing and flexible vNPU-to-pNPU mappings; and (3) an ISA extension of modern NPU architecture for fine-grained tensor operator scheduling for multiple vNPUs. We evaluate Neu10 with a production-level simulator to demonstrate its benefits over state-of-the-art NPU sharing approaches.Keywords
Funding Information
- National Science Foundation (CCF-1919044, CNS-2144796)
This publication has 5 references indexed in Scilit:
- Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms2024
- System Virtualization for Neural Processing Units2023
- V10: Hardware-Assisted NPU Multi-tenancy for Improved Resource Utilization and Fairness2023
- MLPerf Inference Benchmark2020
- PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units2020