Device-Server Collaborative Speculative Decoding for Real-Time LLM Streaming

Authors:
Publisher website
Google Scholar
Abstract
This paper addresses the challenge of real-time text streaming with large language models (LLMs) over unstable networks and server contention. To mitigate delays from packet loss and computational bottlenecks, we propose an adaptive framework that integrates speculative decoding with network-aware optimization. A lightweight draft model runs on the user’s edge device to generate tokens autoregressively, while a larger target model on the server periodically verifies them. We explicitly model end-to-end latency with respect to verification acceptance rates and packet loss, and optimize the verification ratio under latency constraints. Experiments show improved resilience to high packet loss and server contention, achieving over 4× lower rebuffering and 60% higher verification ratios compared to baselines.

This publication has 19 references indexed in Scilit: