Shijie Peng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Shijie Peng has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Kejiang Ye
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
5 shared publications - Yanying Lin
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong, China
5 shared publications - Cheng-Zhong Xu
State Key Laboratory of IoTSC, University of Macau, Macau, Macao
4 shared publications - Chengzhi Lu
Nanyang technological university, Singapore, Singapore
3 shared publications - Shuaipeng Wu
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
3 shared publications - Haiying Shen
Department of Computer science, University of Virginia, Charlottesville, Virginia, USA
1 shared publication
Latest Publications
- Open Access
- 27 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 348-358
ReliefServe: Relieving GPU Pressure in Multi-Model Serving via Selective CPU Escape
- 27 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 348-358
Open AccessReliefServe: Relieving GPU Pressure in Multi-Model Serving via Selective CPU Escape
- Open Access
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 72-88
FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 72-88
Open AccessFlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters
- 13 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (6) , 1442-1457
Workload-Adapted Resource Allocation for LLM Distributed Serving in Serverless Clusters
Large language models increasingly rely on pipeline parallelism for distributed inference, but existing systems face critical challenges in serverless environments: heterogeneous request distributions across pipeline stages and unpredictable workload patterns requiring rapid elasticity. Traditional static resource allocation fails to address pipeline-specific bottlenecks and cold start delays inherent in serverless architectures. We propose Quart, a workload-adapted resource allocation system for LLM distributed serving in serverless clusters. Quart introduces pipeline-aware resource management through: (1) latency-aware critical stage identification using coefficient of variation (CV)-based burst propagation analysis, (2) dynamic replica allocation with proportional-integral-derivative (PID) control for congested stages, and (3) hierarchical parameter caching with copy-on-write mechanisms enabling sub-second serverless scaling. The system addresses serverless-specific challenges through cache-aware scheduling that maintains model parameters in memory, eliminating disk I/O overhead during rapid scaling events. Evaluation with real-world workloads shows Quart reduces average response latency by up to 87.1% compared to existing serverless inference systems while achieving 2.37x improvement in goodput.
- 13 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 37 (6) , 1442-1457
Workload-Adapted Resource Allocation for LLM Distributed Serving in Serverless Clusters
Large language models increasingly rely on pipeline parallelism for distributed inference, but existing systems face critical challenges in serverless environments: heterogeneous request distributions across pipeline stages and unpredictable workload patterns requiring rapid elasticity. Traditional static resource allocation fails to address pipeline-specific bottlenecks and cold start delays inherent in serverless architectures. We propose Quart, a workload-adapted resource allocation system for LLM distributed serving in serverless clusters. Quart introduces pipeline-aware resource management through: (1) latency-aware critical stage identification using coefficient of variation (CV)-based burst propagation analysis, (2) dynamic replica allocation with proportional-integral-derivative (PID) control for congested stages, and (3) hierarchical parameter caching with copy-on-write mechanisms enabling sub-second serverless scaling. The system addresses serverless-specific challenges through cache-aware scheduling that maintains model parameters in memory, eliminating disk I/O overhead during rapid scaling events. Evaluation with real-world workloads shows Quart reduces average response latency by up to 87.1% compared to existing serverless inference systems while achieving 2.37x improvement in goodput.
- 28 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Services Computing
- Vol. 18 (5) , 3164-3176
Serving LLM in Distributed GPU Cluster With Fine-Grain Pipeline Constraints
As Large Language Models (LLMs) continue to advance, their parameter sizes are growing exponentially-far outpacing hardware capabilities. This widening gap necessitates distributed computing through pipeline parallelism for efficient inference. However, the uneven distribution of requests across pipeline stages creates significant performance bottlenecks in real-world deployments. To address this challenge, we present Planck, a performance optimization framework specifically designed for distributed LLM inference. Planck implements fine-grained control through two key mechanisms: a progressive SLO allocation strategy that dynamically adjusts time constraints based on workload patterns, and stage-specific performance controllers that prevent bottlenecks before they cascade through the system. By intelligently balancing resources across pipeline stages, Planck effectively eliminates queue buildup-essentially preventing traffic congestion before it forms. Evaluation using diverse workloads in real cloud environments demonstrates that Planck reduces P99 tail latency by up to 18% and decreases the longest queue lengths by as much as 47.8% across pipeline stages, significantly improving both system responsiveness and resource utilization.
- 28 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Services Computing
- Vol. 18 (5) , 3164-3176
Serving LLM in Distributed GPU Cluster With Fine-Grain Pipeline Constraints
As Large Language Models (LLMs) continue to advance, their parameter sizes are growing exponentially-far outpacing hardware capabilities. This widening gap necessitates distributed computing through pipeline parallelism for efficient inference. However, the uneven distribution of requests across pipeline stages creates significant performance bottlenecks in real-world deployments. To address this challenge, we present Planck, a performance optimization framework specifically designed for distributed LLM inference. Planck implements fine-grained control through two key mechanisms: a progressive SLO allocation strategy that dynamically adjusts time constraints based on workload patterns, and stage-specific performance controllers that prevent bottlenecks before they cascade through the system. By intelligently balancing resources across pipeline stages, Planck effectively eliminates queue buildup-essentially preventing traffic congestion before it forms. Evaluation using diverse workloads in real cloud environments demonstrates that Planck reduces P99 tail latency by up to 18% and decreases the longest queue lengths by as much as 47.8% across pipeline stages, significantly improving both system responsiveness and resource utilization.
- 24 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 13 (1) , 198-212
Understanding Serverless Inference in Mobile-Edge Networks: A Benchmark Approach
Although the emerging serverless paradigm has the potential to become a dominant way of deploying cloud-service tasks across millions of mobile and IoT devices, the overhead characteristics of executing these tasks on such a volume of mobile devices remain largely unclear. To address this issue, this paper conducts a deep analysis based on the OpenFaaS platform—a popular open-source serverless platform for mobile edge environments—to investigate the overhead of performing deep learning inference tasks on mobile devices. To thoroughly evaluate the inference overhead, we develop a performance benchmark, named ESBench, whereby a set of comprehensive experiments are conducted with respect to a bunch of simulated mobile devices associated with an edge cluster. Our investigation reveals that the performance of deep learning inference tasks is significantly influenced by the model size and resource contention in mobile devices, leading to up to 3× degradation in performance. Moreover, we observe that the network environment can negatively impact the performance of mobile inference, increasing the CPU overhead under poor network conditions. Based on our findings, we further propose some recommendations for designing efficient serverless platforms and resource management strategies as well as for deploying serverless computing in the mobile edge environment.
- 24 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 13 (1) , 198-212
Understanding Serverless Inference in Mobile-Edge Networks: A Benchmark Approach
Although the emerging serverless paradigm has the potential to become a dominant way of deploying cloud-service tasks across millions of mobile and IoT devices, the overhead characteristics of executing these tasks on such a volume of mobile devices remain largely unclear. To address this issue, this paper conducts a deep analysis based on the OpenFaaS platform—a popular open-source serverless platform for mobile edge environments—to investigate the overhead of performing deep learning inference tasks on mobile devices. To thoroughly evaluate the inference overhead, we develop a performance benchmark, named ESBench, whereby a set of comprehensive experiments are conducted with respect to a bunch of simulated mobile devices associated with an edge cluster. Our investigation reveals that the performance of deep learning inference tasks is significantly influenced by the model size and resource contention in mobile devices, leading to up to 3× degradation in performance. Moreover, we observe that the network environment can negatively impact the performance of mobile inference, increasing the CPU overhead under poor network conditions. Based on our findings, we further propose some recommendations for designing efficient serverless platforms and resource management strategies as well as for deploying serverless computing in the mobile edge environment.