Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
Authors:
Top Cited Papers
- 29 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1018-1031
Abstract
Large language models (LLMs) based on transformers have made significant strides in recent years, the success of which is driven by scaling up their model size. Despite their high algorithmic performance, the computational and memory requirements of LLMs present unprecedented challenges. To tackle the high compute requirements of LLMs, the Mixture-ofExperts (MoE) architecture was introduced which is able to scale its model size without proportionally scaling up its computational requirements. Unfortunately, MoE’s high memory demands and dynamic activation of sparse experts restrict its applicability to real-world problems. Previous solutions that offload MoE’s memory-hungry expert parameters to CPU memory fall short because the latency to migrate activated experts from CPU to GPU incurs high performance overhead. Our proposed Pre-gated MoE system effectively tackles the compute and memory challenges of conventional MoE architectures using our algorithm-system codesign. Pre-gated MoE employs our novel pre-gating function which alleviates the dynamic nature of sparse expert activation, allowing our proposed system to address the large memory footprint of MoEs while also achieving high performance. We demonstrate that Pre-gated MoE is able to improve performance, reduce GPU memory consumption, while also maintaining the same level of model quality. These features allow our Pre-gated MoE system to cost-effectively deploy large-scale LLMs using just a single GPU with high performance.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale2022
- FasterMoE2022
- MLPerf Inference Benchmark2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- A Configurable Cloud-Scale DNN Processor for Real-Time AI2018
- Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization2018
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Semantic Parsing on Freebase from Question-Answer Pairs2013