Junchen Jiang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Junchen Jiang has a total of 87 co-authors. Here are the top co-authors by number of shared publications:
- Kuntai Du
University of Chicago, Chicago, Illinois, USA
9 shared publications - Yihua Cheng
University of Chicago
7 shared publications - Siddhant Ray
University of Chicago, Chicago, USA
6 shared publications - Yuhan Liu
University of Chicago
6 shared publications - Ganesh Ananthanarayanan
Microsoft, Redmond, USA
4 shared publications - Hanchen Li
University of Chicago
4 shared publications
Latest Publications
- Open Access
- 29 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 27-32
Argo: Efficient Importance Labeling for Enterprise Email Systems
Email importance labeling has been critical for businesses and individuals. However, traditional keyword matching and user-defined rules require manual feature engineering and fail to scale or generalize. Recent LLMs such as GPT-4.1 provide high labeling quality, but enterprise-scale deployment remains prohibitively expensive. We develop Argo, an enterprise-email labeling framework that explores alternative labeling schemes matching GPT-4.1 quality at significantly lower cost, using an efficient profiler to search the cost-quality tradeoff space and identify cost-efficient solutions. Argo also includes an on-demand provisioning scheme to scale labeling with real-time load and minimize peak-load cost increases. Across 3 open-source email datasets, Argo achieves 148-167× inference cost reduction with negligible quality degradation and 20-640000× lower profiling costs, enabling practical, large-scale email labeling for enterprises.
- 29 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 27-32
Open AccessArgo: Efficient Importance Labeling for Enterprise Email Systems
Email importance labeling has been critical for businesses and individuals. However, traditional keyword matching and user-defined rules require manual feature engineering and fail to scale or generalize. Recent LLMs such as GPT-4.1 provide high labeling quality, but enterprise-scale deployment remains prohibitively expensive. We develop Argo, an enterprise-email labeling framework that explores alternative labeling schemes matching GPT-4.1 quality at significantly lower cost, using an efficient profiler to search the cost-quality tradeoff space and identify cost-efficient solutions. Argo also includes an on-demand provisioning scheme to scale labeling with real-time load and minimize peak-load cost increases. Across 3 open-source email datasets, Argo achieves 148-167× inference cost reduction with negligible quality degradation and 20-640000× lower profiling costs, enabling practical, large-scale email labeling for enterprises.
- Top Cited PapersOpen Access
- 23 March 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer Systems
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre -compute the KV cache of a text and re -use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill ( i.e., without reusing KV cache)? This challenge naturally arises in retrieval-augmented generation (RAG) where the input is supplemented with multiple retrieved texts as the context. We present CacheBlend , a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime, the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job, allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on six open-source LLMs of various sizes and five popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2–3.3 × and increases the inference throughput by 2.8-5 × from full KV recompute without compromising generation quality. The code is available at https://github.com/LMCache/LMCache .
- 23 March 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer Systems
Top Cited PapersOpen AccessCacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre -compute the KV cache of a text and re -use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized. This paper tackles just one challenge: when an LLM input contains multiple text chunks, how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill ( i.e., without reusing KV cache)? This challenge naturally arises in retrieval-augmented generation (RAG) where the input is supplemented with multiple retrieved texts as the context. We present CacheBlend , a scheme that reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache. In the meantime, the small extra delay for recomputing some tokens can be pipelined with the retrieval of KV caches within the same job, allowing CacheBlend to store KV caches in slower devices with more storage capacity while retrieving them without increasing the inference delay. By comparing CacheBlend with the state-of-the-art KV cache reusing schemes on six open-source LLMs of various sizes and five popular benchmark datasets of different tasks, we show that CacheBlend reduces time-to-first-token (TTFT) by 2.2–3.3 × and increases the inference throughput by 2.8-5 × from full KV recompute without compromising generation quality. The code is available at https://github.com/LMCache/LMCache .
- Open Access
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 606-622
METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 606-622
Open AccessMETIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
- Open Access
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 399-414
PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 399-414
Open AccessPrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
- Open Access
When to Reason: Semantic Router for vLLM
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also incurs significant costs in inference latency and token usage, with environmental and financial impacts, which are unnecessary for many simple prompts. We present a semantic router that classifies queries based on their reasoning requirements and selectively applies reasoning only when beneficial. Our approach achieves a 10.2 percentage point improvement in accuracy on the MMLU-Pro benchmark while reducing response latency by 47.1% and token consumption by 48.5% compared to direct inference with vLLM. These results demonstrate that semantic routing offers an effective mechanism for striking a balance between accuracy and efficiency in open-source LLM serving systems
Open AccessWhen to Reason: Semantic Router for vLLM
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also incurs significant costs in inference latency and token usage, with environmental and financial impacts, which are unnecessary for many simple prompts. We present a semantic router that classifies queries based on their reasoning requirements and selectively applies reasoning only when beneficial. Our approach achieves a 10.2 percentage point improvement in accuracy on the MMLU-Pro benchmark while reducing response latency by 47.1% and token consumption by 48.5% compared to direct inference with vLLM. These results demonstrate that semantic routing offers an effective mechanism for striking a balance between accuracy and efficiency in open-source LLM serving systems
- Top Cited PapersOpen Access
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 94-109
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 94-109
Top Cited PapersOpen AccessCacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
- Open Access
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 361-376
Earth+: On-Board Satellite Imagery Compression Leveraging Historical Earth Observations
Due to limited downlink (satellite-to-ground) capacity, over 90% of the images captured by the earth-observation satellites are not downloaded to the ground. To overcome the downlink limitation, we present Earth+, a new on-board satellite imagery compression system that identifies and downloads only changed areas in each image compared to latest on-board reference images of the same location. The key of Earth+ is that it obtains latest on-board reference images by letting the ground stations upload images recently captured by all satellites in the constellation. To our best knowledge, Earth+ is the first system that leverages images across an entire satellite constellation to enable more images to be downloaded to the ground (by better satellite imagery compression). Our evaluation shows that to download images of the same area, Earth+ can reduce the downlink usage by 3.3× compared to state-of-the-art on-board image compression techniques without sacrificing imagery quality or using more resources (downlink, computation or storage).
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 361-376
Open AccessEarth+: On-Board Satellite Imagery Compression Leveraging Historical Earth Observations
Due to limited downlink (satellite-to-ground) capacity, over 90% of the images captured by the earth-observation satellites are not downloaded to the ground. To overcome the downlink limitation, we present Earth+, a new on-board satellite imagery compression system that identifies and downloads only changed areas in each image compared to latest on-board reference images of the same location. The key of Earth+ is that it obtains latest on-board reference images by letting the ground stations upload images recently captured by all satellites in the constellation. To our best knowledge, Earth+ is the first system that leverages images across an entire satellite constellation to enable more images to be downloaded to the ground (by better satellite imagery compression). Our evaluation shows that to download images of the same area, Earth+ can reduce the downlink usage by 3.3× compared to state-of-the-art on-board image compression techniques without sacrificing imagery quality or using more resources (downlink, computation or storage).
- Open Access
Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM inputs. This has led to a broad design space, including colocating stored KV caches with (or close to) GPUs to various KV cache compression. However, a key question remains unanswered: can these delay reductions also be economically favorable? Specifically, we ask whether a developer can use public cloud services to store precomputed KV caches and reuse them to save delay without incurring more costs in terms of compute, storage, and network. To answer this question, we propose an validated analytical model for the cloud cost (in compute, storage, and network) of storing and reusing KV caches based on various workload parameters, such as reuse frequency, generated text lengths, model sizes, etc. Preliminary results show that KV cache reusing is able to save both delay and cloud cost across a range of workloads with long context. And we call more efforts on building more economical context augmented LLM by KV cache reusing.
Open AccessTowards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM inputs. This has led to a broad design space, including colocating stored KV caches with (or close to) GPUs to various KV cache compression. However, a key question remains unanswered: can these delay reductions also be economically favorable? Specifically, we ask whether a developer can use public cloud services to store precomputed KV caches and reuse them to save delay without incurring more costs in terms of compute, storage, and network. To answer this question, we propose an validated analytical model for the cloud cost (in compute, storage, and network) of storing and reusing KV caches based on various workload parameters, such as reuse frequency, generated text lengths, model sizes, etc. Preliminary results show that KV cache reusing is able to save both delay and cloud cost across a range of workloads with long context. And we call more efforts on building more economical context augmented LLM by KV cache reusing.
- Open Access
LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.
Open AccessLLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.
- Top Cited PapersOpen Access
- 4 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 38-56
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5--4.3x and the total delay in fetching and processing contexts by 3.2--3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.
- 4 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 38-56
Top Cited PapersOpen AccessCacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5--4.3x and the total delay in fetching and processing contexts by 3.2--3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.