MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
Preprint
Abstract
Mixture of Experts (MoE) models have enabled the scaling of Large Language Models (LLMs) and Vision Language Models (VLMs) by achieving massive parameter counts while maintaining computational efficiency. However, MoEs introduce several inference-time challenges, including load imbalance across experts and the additional routing computational overhead. To address these challenges and fully harness the benefits of MoE, a systematic evaluation of hardware acceleration techniques is essential. We present MoE-Inference-Bench, a comprehensive study to evaluate MoE performance across diverse scenarios. We analyze the impact of batch size, sequence length, and critical MoE hyperparameters such as FFN dimensions and number of experts on throughput. We evaluate several optimization techniques on Nvidia H100 GPUs, including pruning, Fused MoE operations, speculative decoding, quantization, and various parallelization strategies. Our evaluation includes MoEs from the Mixtral, DeepSeek, OLMoE and Qwen families. The results reveal performance differences across configurations and provide insights for the efficient deployment of MoEs.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- A Survey on Inference Optimization Techniques for Mixture of Experts ModelsACM Computing Surveys, 2026
- A Survey on Mixture of Experts in Large Language ModelsIEEE Transactions on Knowledge and Data Engineering, 2025
- AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and AccelerationGetMobile: Mobile Computing and Communications, 2025
- Toward a Holistic Performance Evaluation of Large Language Models Across Diverse AI Accelerators2024
- MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition2024
- Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models2024
- Evaluation of pre-training large language models on leadership-class supercomputersThe Journal of Supercomputing, 2023
- NVIDIA Hopper H100 GPU: Scaling PerformanceIEEE Micro, 2023
- A Comprehensive Evaluation of Novel AI Accelerators for Deep Learning Workloads2022
- Comparative evaluation of deep learning workloads for leadership-class systemsBenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2021