Jinwoo Jeong
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jinwoo Jeong has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Jeongseob Ahn
Korea University, Seoul, South Korea
4 shared publications - Hanhwi Jang
Ajou University, Suwon, South Korea
1 shared publication - Seungsu Baek
Ajou University, Suwon, Republic of Korea
1 shared publication - Woohyung Choi
Ajou University, Suwon, South Korea
1 shared publication
Latest Publications
- Open Access
- 18 January 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer Systems
- Vol. 44 (1) , 1-21
An Efficient DNN Model Serving System using Layer-wise Caching and Direct-Host-Access
With the increasing demand to utilize deep neural networks (DNNs) in online services, it is important to serve DNN models on GPUs in a cost-effective manner. Once the required DNN model is ready in the GPU memory, we can immediately serve the inference requests with low latency. Otherwise, it needs to load the model from host to GPU, adding a significant delay to inference. This article proposes Ignite to minimize cold-start latency while provisioning DL models from host to GPU in server environments. First, we propose LCache to effectively utilize the limited GPU memory for model serving with unique cache replacement policies. We devise layer-wise cache replacement policies tailored for executing DL inferences in a pipelined way. Second, we take advantage of the direct-host-access facility provided by commodity GPUs, allowing access to particular layers of models in the host memory directly from GPU without loading. We show that Ignite can effectively reduce the cold-start latency while increasing the throughput of serving DNN models. When deploying multiple ResNet, BERT, and RoBERTa instances on a DL inference serving system, Ignite shows a significant performance improvement compared to the pipelining technique and stable 99% tail latency.
- 18 January 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer Systems
- Vol. 44 (1) , 1-21
Open AccessAn Efficient DNN Model Serving System using Layer-wise Caching and Direct-Host-Access
With the increasing demand to utilize deep neural networks (DNNs) in online services, it is important to serve DNN models on GPUs in a cost-effective manner. Once the required DNN model is ready in the GPU memory, we can immediately serve the inference requests with low latency. Otherwise, it needs to load the model from host to GPU, adding a significant delay to inference. This article proposes Ignite to minimize cold-start latency while provisioning DL models from host to GPU in server environments. First, we propose LCache to effectively utilize the limited GPU memory for model serving with unique cache replacement policies. We devise layer-wise cache replacement policies tailored for executing DL inferences in a pipelined way. Second, we take advantage of the direct-host-access facility provided by commodity GPUs, allowing access to particular layers of models in the host memory directly from GPU without loading. We show that Ignite can effectively reduce the cold-start latency while increasing the throughput of serving DNN models. When deploying multiple ResNet, BERT, and RoBERTa instances on a DL inference serving system, Ignite shows a significant performance improvement compared to the pipelining technique and stable 99% tail latency.
- Open Access
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessAccelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management
- 23 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (1) , 33-36
GPU-Centric Memory Tiering for LLM Serving With NVIDIA Grace Hopper Superchip
This study investigates the performance of serving large language models (LLMs) with a focus on the highbandwidth interconnect between GPU and CPU using a real NVIDIA Grace Hopper Superchip. This architecture features a GPU-centric memory tiering system, comprising a performance tier with GPU memory and a capacity tier with host memory. We revisit a conventional pipelined execution for LLM inference, utilizing host memory connected via NVLink alongside GPU memory. For the Llama-3.1 8B base (FP16) model, such a GPU-centric tiered memory system meets the target latency requirements for both prefill and decoding while improving throughput compared to the in-memory case, where all model weights are maintained in GPU memory. However, even with NVLink-connected CPU memory, meeting latency constraints for large models like the 70B and 405B FP16 models remains challenging. To address this, we explore the efficacy of model quantization (e.g., AWQ) along with the pipelined execution. Our evaluation reveals that the model quantization makes the pipelined execution a viable solution for serving large models. For the Llama-3.1 70B and 405B AWQ models, we show that the pipelined execution achieves 1.6× and 2.9× throughput improvement, respectively, compared to the inmemory only case, while meeting the latency constraint.
- 23 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (1) , 33-36
GPU-Centric Memory Tiering for LLM Serving With NVIDIA Grace Hopper Superchip
This study investigates the performance of serving large language models (LLMs) with a focus on the highbandwidth interconnect between GPU and CPU using a real NVIDIA Grace Hopper Superchip. This architecture features a GPU-centric memory tiering system, comprising a performance tier with GPU memory and a capacity tier with host memory. We revisit a conventional pipelined execution for LLM inference, utilizing host memory connected via NVLink alongside GPU memory. For the Llama-3.1 8B base (FP16) model, such a GPU-centric tiered memory system meets the target latency requirements for both prefill and decoding while improving throughput compared to the in-memory case, where all model weights are maintained in GPU memory. However, even with NVLink-connected CPU memory, meeting latency constraints for large models like the 70B and 405B FP16 models remains challenging. To address this, we explore the efficacy of model quantization (e.g., AWQ) along with the pipelined execution. Our evaluation reveals that the model quantization makes the pipelined execution a viable solution for serving large models. For the Llama-3.1 70B and 405B AWQ models, we show that the pipelined execution achieves 1.6× and 2.9× throughput improvement, respectively, compared to the inmemory only case, while meeting the latency constraint.
- Open Access
- 8 May 2023
- Published by Association for Computing Machinery (ACM)
- p. 249-265
Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access
As deep learning (DL) inference has been widely adopted for building user-facing applications in many domains, it is increasingly important for DL inference servers to achieve high throughput while preserving bounded latency. DL inference requests can be immediately served if the corresponding model is already in the GPU memory. Otherwise, it needs to load the model from host to GPU, adding a significant delay to inference. This paper proposes DeepPlan to minimize inference latency while provisioning DL models from host to GPU in server environments. First, we take advantage of the direct-host-access facility provided by commodity GPUs, allowing access to particular layers of models in the host memory directly from GPU without loading. Second, we parallelize model transmission across multiple GPUs to reduce the time for loading models from host to GPU. We show that a single inference can achieve a 1.94× speedup compared with the state-of-the-art pipelining approach for BERT-Base. When deploying multiple BERT, RoBERTa, and GPT-2 instances on a DL inference serving system, DeepPlan shows a significant performance improvement compared to the pipelining technique and stable 99% tail latency.
- 8 May 2023
- Published by Association for Computing Machinery (ACM)
- p. 249-265
Open AccessFast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access
As deep learning (DL) inference has been widely adopted for building user-facing applications in many domains, it is increasingly important for DL inference servers to achieve high throughput while preserving bounded latency. DL inference requests can be immediately served if the corresponding model is already in the GPU memory. Otherwise, it needs to load the model from host to GPU, adding a significant delay to inference. This paper proposes DeepPlan to minimize inference latency while provisioning DL models from host to GPU in server environments. First, we take advantage of the direct-host-access facility provided by commodity GPUs, allowing access to particular layers of models in the host memory directly from GPU without loading. Second, we parallelize model transmission across multiple GPUs to reduce the time for loading models from host to GPU. We show that a single inference can achieve a 1.94× speedup compared with the state-of-the-art pipelining approach for BERT-Base. When deploying multiple BERT, RoBERTa, and GPT-2 instances on a DL inference serving system, DeepPlan shows a significant performance improvement compared to the pipelining technique and stable 99% tail latency.