FlexSpec: Frozen Drafts Meet Evolving Targets in Edge-Cloud Collaborative LLM Speculative Decoding

Authors:
  • YLYuchen Li
    Yuchen Li
  • RKRui Kong
    Rui Kong
  • ZLZhonghao Lyu
    Zhonghao Lyu
  • QLQiyang Li
    Qiyang Li
  • XCXinran Chen
    Xinran Chen
  • HCHengyi Cai
    Hengyi Cai
  • LYLingyong Yan
    Lingyong Yan
  • SWShuaiqiang Wang
    Shuaiqiang Wang
  • LKLinghe Kong
    Linghe Kong
  • GCGuihai Chen
    Guihai Chen
Preprint
Publisher website
Google Scholar
Abstract
Deploying large language models (LLMs) in mobile and edge computing environments is constrained by limited on-device resources, scarce wireless bandwidth, and frequent model evolution. Although edge-cloud collaborative inference with speculative decoding (SD) can reduce end-to-end latency by executing a lightweight draft model at the edge and verifying it with a cloud-side target model, existing frameworks fundamentally rely on tight coupling between the two models. Consequently, repeated model synchronization introduces excessive communication overhead, increasing end-to-end latency, and ultimately limiting the scalability of SD in edge environments. To address these limitations, we propose FlexSpec, a communication-efficient collaborative inference framework tailored for evolving edge-cloud systems. The core design of FlexSpec is a shared-backbone architecture that allows a single and static edge-side draft model to remain compatible with a large family of evolving cloud-side target models. By decoupling edge deployment from cloud-side model updates, FlexSpec eliminates the need for edge-side retraining or repeated model downloads, substantially reducing communication and maintenance costs. Furthermore, to accommodate time-varying wireless conditions and heterogeneous device constraints, we develop a channel-aware adaptive speculation mechanism that dynamically adjusts the speculative draft length based on real-time channel state information and device energy budgets. Extensive experiments demonstrate that FlexSpec achieves superior performance compared to conventional SD approaches in terms of inference efficiency.
All Related Versions

This publication has 9 references indexed in Scilit: