Software-Hardware Co-design of Heterogeneous SmartNIC System for Recommendation Models Inference and Training
Open Access
- 21 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 336-347
Abstract
Deep Learning Recommendation Models (DLRMs) are important applications in various domains and have evolved into one of the largest and most important machine learning applications. With their trillions of parameters necessarily exceeding the high bandwidth memory (HBM) capacity of GPUs, ever more massive DLRMs require large-scale multi-node systems for distributed training and inference. However, these all suffer from the all-to-all communication bottleneck, which limits scalability. SmartNICs couple computation and communication capabilities to provide powerful network-facing heterogeneous devices that reduce communication overhead. There has not, however, been a distributed system design that fully leverages SmartNIC resources to address scalability of DLRMs. We propose a software-hardware co-design of a heterogeneous SmartNIC system that overcomes the communication bottleneck of distributed DLRMs, mitigates the pressure on memory bandwidth, and improves computation efficiency. We provide a set of SmartNIC designs of cache systems (including local cache and remote cache) and SmartNIC computation kernels that reduce data movement, relieve memory lookup intensity, and improve the GPU's computation efficiency. In addition, we propose a graph algorithm that improves the data locality of queries within batches and optimizes the overall system performance with higher data reuse. Our evaluation shows that the system achieves 2.1× latency speedup for inference and 1.6× throughput speedup for training.Keywords
This publication has 35 references indexed in Scilit:
- Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems2020
- COnfigurable Network Protocol Accelerator (COPA): An Integrated Networking/Accelerator Hardware/Software Framework2020
- INCA2019
- Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective2018
- sPIN2017
- Two Decades of Recommender Systems at Amazon.comIEEE Internet Computing, 2017
- A cloud-scale acceleration architecture2016
- Deep Neural Networks for YouTube Recommendations2016
- The Netflix Recommender SystemACM Transactions on Management Information Systems, 2015
- A configurable network protocol for cluster based communications using modular hardware primitives on an intelligent NIC2003