SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency
- 24 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (2) , 265-268
Abstract
Large Language Models (LLMs) applying Mixture-of-Experts (MoE) scale to trillions of parameters but require vast memory, motivating a line of research to offload expert weights from fast-but-small DRAM (HBM) to denser Flash SSDs. While SSDs provide cost-effective capacity, their read energy per bit is substantially higher than that of DRAM. This paper quantitatively analyzes the energy implications of offloading MoE expert weights to SSDs during the critical decode stage of LLM inference. Our analysis, comparing SSD, CPU memory (DDR), and HBM storage scenarios for models like DeepSeek-R1, reveals that offloading MoE weights to current SSDs drastically increases per-token-generation energy consumption (e.g., by up to $\sim 12\times$ compared to the HBM baseline), dominating the total inference energy budget. Although techniques like prefetching effectively hide access latency, they cannot mitigate this fundamental energy penalty. We further explore future technological scaling, finding that the inherent sparsity of MoE models could potentially make SSDs energy-viable if Flash read energy improves significantly, roughly by an order of magnitude.Keywords
All Related Versions
Funding Information
- Mobile eXperience (MX) Business, Samsung Electronics Company Ltd.
- Ministry of Science, ICT (RS-2024-00456287)
- Institute for Information & Communications Technology Planning
This publication has 6 references indexed in Scilit:
- Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching2024
- Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference2024
- LLM in a flash: Efficient Large Language Model Inference with Limited Memory2024
- A 135 GBps/Gbit 0.66 pJ/bit Stacked Embedded DRAM with Multilayer Arrays by Fine Pitch Hybrid Bonding and Mini-TSV2023
- Fine-grained DRAM2017
- FlashPower: A detailed power model for NAND flash memory2010