Haiying Shen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Haiying Shen has a total of 48 co-authors. Here are the top co-authors by number of shared publications:
- Zeyu Zhang
Department of Computer science, University of Virginia, Charlottesville, Virginia, USA
4 shared publications - Sudipta Saha Shubha
University of Virginia, Charlottesville, USA
3 shared publications - Anand Iyer
Georgia Institute of Technology, Atlanta, Georgia, USA
2 shared publications - Cheng-Zhong Xu
State Key Laboratory of IoTSC, University of Macau, Macau, Macao
2 shared publications - Kejiang Ye
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
2 shared publications - Michael Mitzenmacher
Harvard University, Cambridge, MA, USA
2 shared publications
Latest Publications
- Open Access
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 754-770
HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 754-770
Open AccessHACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
- Open Access
- 27 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 208-217
Heterogeneous SLO Guaranteed Multi-Resource-Aware Batching in LLM Serving
In this paper, we study a mixed-prompt scenario—where both short and long prompts coexist—in an LLM inference serving system that supports diverse applications with heterogeneous iteration-time SLOs. To improve throughput for long prompts, prior work divides them into chunks and batches requests or chunks to meet the token budget to fully utilize GPU compute resource. Our measurement analysis shows that it fails to meet heterogeneous iteration-time SLOs or fully utilize both GPU compute and memory resources. Also, delayed KV cache release from processing multiple long prompts may reduce throughput. To address the limitations, based on our findings, we propose a heterogeneous SLO guaranteed Multi-Resource-aware batching system (MuRa). MuRa orders queued requests by SLOs, enabling it to prioritize tight-SLO requests and batch those with similar SLOs to improve throughput. It selectively chooses queued requests to maximize both compute and memory utilizations, and processes all chunks from one prompt before moving to the next to enable earlier KV cache release. Trace-driven real experiments demonstrate that MuRa achieves 1.42-11.21 × higher throughput, 1.43-13.71 × higher goodput, 37-90% higher SLO attainment, and 1.61-12.22 × lower response latency compared to the state-of-the-art approaches. It achieves performance near Oracle, which optimally maximizes goodput.
- 27 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 208-217
Open AccessHeterogeneous SLO Guaranteed Multi-Resource-Aware Batching in LLM Serving
In this paper, we study a mixed-prompt scenario—where both short and long prompts coexist—in an LLM inference serving system that supports diverse applications with heterogeneous iteration-time SLOs. To improve throughput for long prompts, prior work divides them into chunks and batches requests or chunks to meet the token budget to fully utilize GPU compute resource. Our measurement analysis shows that it fails to meet heterogeneous iteration-time SLOs or fully utilize both GPU compute and memory resources. Also, delayed KV cache release from processing multiple long prompts may reduce throughput. To address the limitations, based on our findings, we propose a heterogeneous SLO guaranteed Multi-Resource-aware batching system (MuRa). MuRa orders queued requests by SLOs, enabling it to prioritize tight-SLO requests and batch those with similar SLOs to improve throughput. It selectively chooses queued requests to maximize both compute and memory utilizations, and processes all chunks from one prompt before moving to the next to enable earlier KV cache release. Trace-driven real experiments demonstrate that MuRa achieves 1.42-11.21 × higher throughput, 1.43-13.71 × higher goodput, 37-90% higher SLO attainment, and 1.61-12.22 × lower response latency compared to the state-of-the-art approaches. It achieves performance near Oracle, which optimally maximizes goodput.
- Open Access
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1111-1127
AdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1111-1127
Open AccessAdaGen: Workload-Adaptive Cluster Scheduler for Latency-Optimal LLM Inference Serving
- Open Access
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1862-1878
AIMS: Cost-Efficient LLM-Based Agent Deployment in Hybrid Cloud-Edge Environments
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1862-1878
Open AccessAIMS: Cost-Efficient LLM-Based Agent Deployment in Hybrid Cloud-Edge Environments
- 13 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (6) , 1442-1457
Workload-Adapted Resource Allocation for LLM Distributed Serving in Serverless Clusters
Large language models increasingly rely on pipeline parallelism for distributed inference, but existing systems face critical challenges in serverless environments: heterogeneous request distributions across pipeline stages and unpredictable workload patterns requiring rapid elasticity. Traditional static resource allocation fails to address pipeline-specific bottlenecks and cold start delays inherent in serverless architectures. We propose Quart, a workload-adapted resource allocation system for LLM distributed serving in serverless clusters. Quart introduces pipeline-aware resource management through: (1) latency-aware critical stage identification using coefficient of variation (CV)-based burst propagation analysis, (2) dynamic replica allocation with proportional-integral-derivative (PID) control for congested stages, and (3) hierarchical parameter caching with copy-on-write mechanisms enabling sub-second serverless scaling. The system addresses serverless-specific challenges through cache-aware scheduling that maintains model parameters in memory, eliminating disk I/O overhead during rapid scaling events. Evaluation with real-world workloads shows Quart reduces average response latency by up to 87.1% compared to existing serverless inference systems while achieving 2.37x improvement in goodput.
- 13 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (6) , 1442-1457
Workload-Adapted Resource Allocation for LLM Distributed Serving in Serverless Clusters
Large language models increasingly rely on pipeline parallelism for distributed inference, but existing systems face critical challenges in serverless environments: heterogeneous request distributions across pipeline stages and unpredictable workload patterns requiring rapid elasticity. Traditional static resource allocation fails to address pipeline-specific bottlenecks and cold start delays inherent in serverless architectures. We propose Quart, a workload-adapted resource allocation system for LLM distributed serving in serverless clusters. Quart introduces pipeline-aware resource management through: (1) latency-aware critical stage identification using coefficient of variation (CV)-based burst propagation analysis, (2) dynamic replica allocation with proportional-integral-derivative (PID) control for congested stages, and (3) hierarchical parameter caching with copy-on-write mechanisms enabling sub-second serverless scaling. The system addresses serverless-specific challenges through cache-aware scheduling that maintains model parameters in memory, eliminating disk I/O overhead during rapid scaling events. Evaluation with real-world workloads shows Quart reduces average response latency by up to 87.1% compared to existing serverless inference systems while achieving 2.37x improvement in goodput.
- Open Access
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 817-830
M od S erve : Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 817-830
Open AccessM od S erve : Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
- Open Access
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 600-613
CIS: Checkpointed Inference for Data Drift-Resilient Model Serving at Edge Servers
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 600-613
Open AccessCIS: Checkpointed Inference for Data Drift-Resilient Model Serving at Edge Servers
- Open Access
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Understanding Diffusion Model Serving in Production: A Top-Down Analysis of Workload, Scheduling, and Resource Efficiency
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessUnderstanding Diffusion Model Serving in Production: A Top-Down Analysis of Workload, Scheduling, and Resource Efficiency
- Open Access
- 27 August 2025
- Published by Association for Computing Machinery (ACM)
- p. 1245-1247
HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
- 27 August 2025
- Published by Association for Computing Machinery (ACM)
- p. 1245-1247
Open AccessHACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
- Open Access
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 147-161
Concurrency-Informed Orchestration for Serverless Functions
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 147-161
Open AccessConcurrency-Informed Orchestration for Serverless Functions