QZQizheng Zhang

Qizheng Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Qizheng Zhang has a total of 28 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers
    Open Access

    CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

    by Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang

    Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre -compute the KV cache of a text and re -use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill ( i.e., without reusing KV cache)? This challenge naturally arises in retrieval-augmented generation (RAG) where the input is supplemented with multiple retrieved texts as the context. We present CacheBlend , a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime, the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job, allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on six open-source LLMs of various sizes and five popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2–3.3 × and increases the inference throughput by 2.8-5 × from full KV recompute without compromising generation quality. The code is available at https://github.com/LMCache/LMCache .


  • Top Cited Papers
    Open Access

    CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

    by Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang


  • Open Access

    Caravan: Practical Online Learning of In-Network ML Models with Labeling Agents

    by Qizheng Zhang, Ali Imran, Enkeleda Bardhi, Tushar Swamy, Nathan Zhang, Muhammad Shahbaz, Kunle Olukotun

    Recent work on in-network machine learning (ML) anticipates offline models to operate well in modern networking environments. However, upon deployment, these models struggle to cope with fluctuating traffic patterns and network conditions and, therefore, must be validated and updated frequently in an online fashion. This abstract presents Caravan, a practical online learning system for in-network ML models. We tackle two primary challenges in facilitating online learning for networking: (a) the automatic labeling of evolving traffic and (b) the efficient monitoring and detection of model performance degradation to trigger retraining. Caravan repurposes existing systems (e.g., heuristics, access control lists, and foundation models)---not directly suitable for such dynamic environments---into high-quality labeling sources for generating labeled data for online learning. Caravan also introduces a new metric, accuracy proxy, to track model degradation and potential drift to efficiently trigger retraining. Our evaluations show that Caravan's labeling strategy enables in-network ML models to closely follow the changes in the traffic dynamics with a 30.3% improvement in F1 score (on average), compared to offline models. Moreover, Caravan sustains comparable inference accuracy to that of a continuous-learning system while consuming 61.3% less GPU compute time (on average) via accuracy proxy and retraining triggers. This abstract summarizes a previously published work [53] at the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24).


  • Top Cited Papers
    Open Access

    CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

    by Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al.

    As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5--4.3x and the total delay in fetching and processing contexts by 3.2--3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.


  • Open Access

    OneAdapt

    by Kuntai Du, Yuhan Liu, Yitian Hao, Qizheng Zhang, Haodong Wang, Yuyang Huang, Ganesh Ananthanarayanan, Junchen Jiang


  • Open Access

    Optimizing Real-Time Video Experience with Data Scalable Codec

    by Hanchen Li, Yihua Cheng, Ziyi Zhang, Qizheng Zhang, Anton Arapin, Nick Feamster, Amrita Mazumdar

    Real-time video communication is becoming more and more important. However, packet loss is prevalent and resending packets, especially in long-latency networks, causes visual stalls. Previous solutions all perform suboptimally as they either add redundancy before sending the data, which reduces bitrate when no packet is lost, or fail to prevent video freeze when redundancy is not enough. User studies confirm that both bitrate decrease and video freeze significantly damage users' Quality of Experience (QoE). Through a user study comparing different artifacts during a quality drop period, we find that moderate quality drop is preferred over video freeze during packet loss. Inspired by this, we propose a new solution that trains a neural network Autoencoder to optimize frame quality under different packet loss rates. Our insight is that such training produces a Data Scalable codec, whose quality increases with each new packet arrival and reaches highest quality when no packet is lost. Specifically, with the arrival of any x encoded bytes of a frame, the decoded quality is closer to the quality than if the whole frame were encoded with x bytes in the first place. Thus, unless all packets are lost, our approach causes a moderate quality drop instead of video freeze during packet loss. In the end, we identify the technical challenges remaining in this approach and point out future opportunities.


  • Application of Thompson sampling on multi-target search

    by Zhaolin Wang, Yangyang Gao, Qirui Xiao, Qizheng Zhang

    Multi-robot coordination demonstrates outstanding performance in exploring and tracking targets in unknown environments, especially in complex terrain or disaster environments. In these cases, the distribution of the search area is changeable and irregular, and the exploration target may also move in an unknown environment. To quickly determine the location of these exploration targets, we propose a distributed control strategy in the article, which applies Bernoulli Thompson sampling to solve the multi-armed bandit problem to allow multiple robots to share information and efficiently search targets. This algorithm significantly speeds up the robot's search for targets, especially when most targets are concentrated in certain fixed areas of the environment.