Abstract
Mamba has emerged as a promising alternative to Transformers for on-device LLM inference, offering linear complexity and eliminating KV-cache. However, autoregressive decoding reloads full model weights every token, accounting for 93.2% of total inference energy, with no existing accelerator resolving this overhead. We present HotBa, a heterogeneous Mamba accelerator that reduces per-token weight transfer by 82% and redundant computation by 59% via Δ-guided early rejection for wide-tree speculative decoding in Mamba, while a heterogeneous INT8/FP16 core and tree management unit achieve 40.4× area efficiency and 5.18× SSM speedup with 0.4% area overhead. Synthesized in 28nm CMOS, HotBa achieves 75.32 tokens/s with 1.65× speedup and 7.55× energy efficiency over the state-of-the-art Mamba accelerator.
Funding Information
  • Institute of Information & Communications Technology Planning & Evaluation (IITP) / Korea government (MSIT) (RS-2025-09942968)
  • Institute of Information & Communications Technology Planning & Evaluation (IITP) / Korea government (MSIT) (RS-2024-00399817)
  • IC Design Education Center (IDEC), Korea (EDA tool)