Herald: An Embedding Scheduler for Distributed Embedding Model Training
Open Access
- 1 July 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 50-56
Abstract
Given the ability to represent categorical features, embedding models have gained great success on many internet services. State-of-the-art training frameworks enable embedding cache in GPU workers to benefit from hardware acceleration while supporting massive category representations (embeddings) in the limited-capacity GPU device memory. However, based on our measurements, naively adopting a cache system in embedding model training leads to non-negligible communications overhead between caches and the global parameter server. We observe that many such communications are avoidable, given the predictability and sparsity natures of embedding cache accesses in distributed training. In this paper, we propose Herald, a runtime embedding scheduler that significantly reduces the cache overhead by leveraging information about the required embeddings in the input samples and the locations of those embeddings. Herald is composed of two key optimizations: It allocates samples in a training batch to proper workers for a high cache hit rate via a heuristic location-aware inputs partition mechanism, and applies an on-demand synchronization strategy for a low frequency of embedding synchronization. Preliminary simulation results show that Herald can reduce cache overhead by 39.3%-53.7% compared to a naive cache-enabled training system across different realistic datasets.Keywords
Funding Information
- the Hong Kong RGC TRS T41-603/20-R (GRF 16213621, GRF 16215119)
- the Key-Area Research and Development Program of Guangdong Province (2021B0101400001)
This publication has 17 references indexed in Scilit:
- SparCML2019
- XDL2019
- MOBIUS2019
- Deep Interest Evolution Network for Click-Through Rate PredictionProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Parallax2019
- Reacting to Variations in Product Demand: An Application for Conversion Rate (CR) Prediction in Sponsored Search2018
- Deep Interest Network for Click-Through Rate Prediction2018
- Wide & Deep Learning for Recommender Systems2016
- Scaling Distributed Machine Learning with the Parameter Server2014
- OpenMP: an industry standard API for shared-memory programmingIEEE Computational Science and Engineering, 1998