Abstract
Large language models (LLMs) are widely deployed as interactive neural inference services, where performance depends on how compute, memory, communication bandwidth, and persistent state are allocated under non-stationary demand. This survey studies load balancing for LLM inference serving as an intelligent scheduling and resource-allocation problem for large-scale learning systems. The analysis focuses on imbalance induced by prefill–decode phase asymmetry, key–value (KV) cache residency, heavy-tailed request variability, heterogeneous hardware, and service-level objectives (SLOs). A structured review protocol codes representative studies by imbalance source, observable signals, control knobs, assumptions, limitations, and evaluation evidence. We introduce a state– action–constraint abstraction, compare six strategy families, and synthesize evaluation practice around tail latency, time to first token (TTFT), token cadence, goodput, robustness, and energy per SLO-compliant token. Open directions are discussed for long-context and multimodal serving, decentralized deployment, green inference, and autonomous self-adaptive serving.