Woohyung Choi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Woohyung Choi has a total of 3 co-authors. Here are the top co-authors by number of shared publications:
- Hanhwi Jang
Ajou University, Suwon, South Korea
1 shared publication - Jeongseob Ahn
Korea University, Seoul, South Korea
1 shared publication - Jinwoo Jeong
Korea University
1 shared publication
Latest Publications
- 23 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (1) , 33-36
GPU-Centric Memory Tiering for LLM Serving With NVIDIA Grace Hopper Superchip
This study investigates the performance of serving large language models (LLMs) with a focus on the highbandwidth interconnect between GPU and CPU using a real NVIDIA Grace Hopper Superchip. This architecture features a GPU-centric memory tiering system, comprising a performance tier with GPU memory and a capacity tier with host memory. We revisit a conventional pipelined execution for LLM inference, utilizing host memory connected via NVLink alongside GPU memory. For the Llama-3.1 8B base (FP16) model, such a GPU-centric tiered memory system meets the target latency requirements for both prefill and decoding while improving throughput compared to the in-memory case, where all model weights are maintained in GPU memory. However, even with NVLink-connected CPU memory, meeting latency constraints for large models like the 70B and 405B FP16 models remains challenging. To address this, we explore the efficacy of model quantization (e.g., AWQ) along with the pipelined execution. Our evaluation reveals that the model quantization makes the pipelined execution a viable solution for serving large models. For the Llama-3.1 70B and 405B AWQ models, we show that the pipelined execution achieves 1.6× and 2.9× throughput improvement, respectively, compared to the inmemory only case, while meeting the latency constraint.
- 23 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (1) , 33-36
GPU-Centric Memory Tiering for LLM Serving With NVIDIA Grace Hopper Superchip
This study investigates the performance of serving large language models (LLMs) with a focus on the highbandwidth interconnect between GPU and CPU using a real NVIDIA Grace Hopper Superchip. This architecture features a GPU-centric memory tiering system, comprising a performance tier with GPU memory and a capacity tier with host memory. We revisit a conventional pipelined execution for LLM inference, utilizing host memory connected via NVLink alongside GPU memory. For the Llama-3.1 8B base (FP16) model, such a GPU-centric tiered memory system meets the target latency requirements for both prefill and decoding while improving throughput compared to the in-memory case, where all model weights are maintained in GPU memory. However, even with NVLink-connected CPU memory, meeting latency constraints for large models like the 70B and 405B FP16 models remains challenging. To address this, we explore the efficacy of model quantization (e.g., AWQ) along with the pipelined execution. Our evaluation reveals that the model quantization makes the pipelined execution a viable solution for serving large models. For the Llama-3.1 70B and 405B AWQ models, we show that the pipelined execution achieves 1.6× and 2.9× throughput improvement, respectively, compared to the inmemory only case, while meeting the latency constraint.