JHJian Huang

Jian Huang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Jian Huang has a total of 68 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Managing Scalable Direct Storage Accesses for GPUs with GoFS

    by Shaobo Li, Yirui Eric Zhou, Yuqi Xue, Yuan Xu, Jian Huang


  • Open Access

    Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage

    by Ziqi Yuan, Haoyang Zhang, Yirui Eric Zhou, Apoorve Mohan, I-Hsin Chung, Seetharami Seelam, Jian Huang

    We present the design and implementation of a new lifetime-aware tensor offloading framework for GPU memory expansion using low-cost PCIe-based solid-state drives (SSDs). Our framework, TERAIO, is developed explicitly for large language model (LLM) training with multiple GPUs and multiple SSDs. Its design is driven by our observation that the active tensors take only a small fraction (1.7% on average) of allocated GPU memory in each LLM training iteration, the inactive tensors are usually large and will not be used for a long period of time, creating ample opportunities for offloading/prefetching tensors to/from slow SSDs without stalling the GPU training process. TERAIO accurately estimates the lifetime (active period of time in GPU memory) of each tensor with the profiling of the first few iterations in the training process. With the tensor lifetime analysis, TERAIO will generate an optimized tensor offloading/prefetching plan and integrate it into the compiled LLM program via PyTorch. TERAIO has a runtime tensor migration engine to execute the offloading/prefetching plan via GPUDirect storage, which allows direct tensor migration between GPUs and SSDs for alleviating the CPU bottleneck and maximizing the SSD bandwidth utilization. In comparison with state-of-the-art studies such as ZeRO-Offload and ZeRO-Infinity, we show that TERAIO improves the training performance of various LLMs by 1.47x on average, and achieves 80.7% of the ideal performance assuming unlimited GPU memory.

  • Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms

    by Yuqi Xue, Yiqi Liu, Lifeng Nai, Jian Huang

    Cloud platforms have deployed hardware accelerators like neural processing units (NPUs) for machine learning (ML) inference services. To maximize resource utilization while ensuring quality of service, a natural approach is to virtualize NPUs for efficient resource sharing for multi-tenant ML services. However, virtualizing NPUs is challenging. This is not only due to the lack of system abstraction support for NPU hardware, but also due to the lack of architectural and ISA support for fine-grained operator scheduling. This article presents Neu10, an NPU virtualization framework consisting of (1) an abstraction called vNPU for fine-grained virtualization of the heterogeneous compute units in a physical NPU (pNPU); (2) a vNPU allocator that enables pay-as-you-go pricing and flexible vNPU-to-pNPU mappings; and (3) an ISA extension of modern NPU architecture for fine-grained tensor operator scheduling for multiple vNPUs. We evaluate Neu10 with a production-level simulator to demonstrate its benefits over state-of-the-art NPU sharing approaches.


  • Open Access

    Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms

    by Benjamin Reidys, Pantea Zardoshti, Íñigo Goiri, Celine Irvene, Daniel S. Berger, Haoran Ma, Kapil Arya, Eli Cortez, Taylor Stark, Eugene Bak, et al.

    Cloud platforms remain underutilized despite multiple proposals to improve their utilization (e.g., disaggregation, harvesting, and oversubscription). Our characterization of the resource utilization of virtual machines (VMs) in Azure reveals that, while CPU is the main underutilized resource, we need to provide a solution to manage all resources holistically. We also observe that many VMs exhibit complementary temporal patterns, which can be leveraged to improve the oversubscription of underutilized resources. Based on these insights, we propose Coach: a system that exploits temporal patterns for all-resource oversubscription in cloud platforms. Coach uses long-term predictions and an efficient VM scheduling policy to exploit temporally complementary patterns. We introduce a new general-purpose VM type, called CoachVM, where we partition each resource allocation into a guaranteed and an oversubscribed portion. Coach monitors the oversubscribed resources to detect contention and mitigate any potential performance degradation. We focus on memory management, which is particularly challenging due to memory's sensitivity to contention and the overhead required to reassign it between CoachVMs. Our experiments show that Coach enables platforms to host up to ~26% more VMs with minimal performance degradation.


  • Open Access

    ByteFS: System Support for (CXL-based) Memory-Semantic Solid-State Drives

    by Shaobo Li, Yirui (Eric) Zhou, Hao Ren, Jian Huang

    Unlike non-volatile memory that resides on the processor memory bus, memory-semantic solid-state drives (SSDs) support both byte and block access granularity via PCIe or CXL interconnects. They provide scalable memory capacity using NAND flash at a much lower cost. In addition, they have different performance characteristics for their dual byte/block interface respectively, while offering essential memory semantics for upper-level software. Such a byte-accessible storage device provides new implications on the software system design. In this paper, we develop a new file system, named ByteFS, by rethinking the design primitives of file systems and SSD firmware to exploit the advantages of both byte and block-granular data accesses. ByteFS supports byte-granular data persistence to retain the persistence nature of SSDs. It extends the core data structure of file systems by enabling dual byte/block-granular data accesses. To facilitate the support for byte-granular writes, ByteFS manages the internal DRAM of SSD firmware in a log-structured manner and enables data coalescing to reduce the unnecessary I/O traffic to flash chips. ByteFS also enables coordinated data caching between the host page cache and SSD cache for best utilizing the precious memory resource. We implement ByteFS on both a real programmable SSD and an emulated memory-semantic SSD for sensitivity study. Compared to state-of-the-art file systems for non-volatile memory and conventional SSDs, ByteFS outperforms them by up to 2.7×, while preserving the essential properties of a file system. ByteFS also reduces the write traffic to SSDs by up to 5.1× by alleviating unnecessary writes caused by both metadata and data updates in file systems.


  • Open Access

    FleetIO: Managing Multi-Tenant Cloud Storage with Multi-Agent Reinforcement Learning

    by Jinghan Sun, Benjamin Reidys, Daixuan Li, Jichuan Chang, Marc Snir, Jian Huang

    Cloud platforms have been virtualizing storage devices like flash-based solid-state drives (SSDs) to make effective use of storage resources. They enable either software-isolated instance or hardware-isolated instance for facilitating the storage sharing between multi-tenant applications. However, for decades, they have to combat the fundamental tussle between the performance isolation and resource utilization. They suffer from either long tail latency caused by weak isolation or low storage utilization caused by strong isolation. In this paper, we present FleetIO, a learning-based storage virtualization framework that employs reinforcement learning (RL) for managing virtualized SSDs. FleetIO explores the unique features of RL to handle the dynamic changes of application workloads and storage states, and integrates the storage scheduling into the RL decision-making process. It achieves both performance isolation and improved storage utilization by enabling dynamic fine-grained storage harvesting across collocated application instances, while minimizing its negative impact on their service-level objectives (SLOs). FleetIO clusters workloads into different types (e.g., latency-sensitive and bandwidth-intensive) based on the collected I/O traces at runtime, and fine-tunes the RL reward functions for each type of workloads. We implement FleetIO on a real programmable SSD board and evaluate it with diverse cloud applications. We show that FleetIO improves the overall storage utilization of the shared SSD by up to 1.4×, and decreases the tail latency of I/O requests by 1.5× on average, compared to the state-of-the-art storage sharing approaches.


  • Open Access

    Exploring the Efficiency of Renewable Energy-based Modular Data Centers at Scale

    by Jinghan Sun, Zibo Gong, Anup Agarwal, Shadi Noghabi, Ranveer Chandra, Marc Snir, Jian Huang

    Modular data centers (MDCs) that can be placed right at the energy farms and powered mostly by renewable energy, is a flexible and effective approach to lowering the carbon footprint of data centers. However, the main challenge of using renewable energy is the high variability of power produced, which implies large volatility in powering computing resources at MDCs, and degraded application performance due to the task evictions and migrations. This causes challenges for platform operators to decide the MDC deployment. To this end, we present SkyBox, a framework that employs a learning-based approach for platform operators to explore the efficient use of renewable energy with MDC deployment across geographical regions. SkyBox is driven by the insights based on our study of real-world power traces from a variety of renewable energy farms - the predictable production of renewable energy and the complementary nature of energy production patterns across different renewable energy sources and locations. With these insights, SkyBox uses the coefficient of variation metric to select the qualified renewable farms, it can identify a set of farms with complementary energy production patterns with a subgraph identification algorithm. After that, SkyBox enables smart workload placement and migrations to further tolerate the power variability. Our experiments with real power traces and datacenter workloads show that SkyBox has the lowest carbon emissions compared with existing approaches. SkyBox also minimizes the negative impact of the power variability on cloud applications, enabling it an effective solution of utilizing renewable energy for modern data centers.


  • Open Access

    Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10

    by Yiqi Liu, Yuqi Xue, Yu Cheng, Lingxiao Ma, Ziming Miao, Jilong Xue, Jian Huang

    As AI chips incorporate numerous parallelized cores to scale deep learning (DL) computing, inter-core communication is enabled recently by employing high-bandwidth and low-latency interconnect links on the chip (e.g., Graphcore IPU). It allows each core to directly access the fast scratchpad memory in other cores, which enables new parallel computing paradigms. However, without proper support for the scalable inter-core connections in current DL compilers, it is hard for developers to exploit the benefits of this new architecture. We present T10, the first DL compiler to exploit the inter-core communication bandwidth and distributed on-chip memory on AI chips. To formulate the computation and communication patterns of tensor operators in this new architecture, T10 introduces a distributed tensor abstraction rTensor. T10 maps a DNN model to execution plans with a generalized compute-shift pattern, by partitioning DNN computation into sub-operators and mapping them to cores, so that the cores can exchange data following predictable patterns. T10 makes globally optimized trade-offs between on-chip memory consumption and inter-core communication overhead, selects the best execution plan from a vast optimization space, and alleviates unnecessary inter-core communications. Our evaluation with a real inter-core connected AI chip, the Graphcore IPU, shows up to 3.3× performance improvement, and scalability support for larger models, compared to state-of-the-art DL compilers and vendor libraries.