Chengtao Lai
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Chengtao Lai has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Wei Zhang
The Hong Kong University of Science and Technology
5 shared publications - Zhongchun Zhou
The Hong Kong University of Science and Technology
4 shared publications - Yuhang Gu
The Hong Kong University of Science and Technology
2 shared publications - Akash Poptani
Indian Institute of Technology Dharwad, INDIA
1 shared publication
Latest Publications
- 1 December 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computers
- Vol. 75 (12) , 4460-4474
DCO: Dynamic Cache Orchestration for LLM Accelerators Through Predictive Management
- 1 December 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computers
- Vol. 75 (12) , 4460-4474
DCO: Dynamic Cache Orchestration for LLM Accelerators Through Predictive Management
- Open Access
DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management
The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and their asynchronous management, we investigate the opposite point of the design spectrum: a multi-core AI accelerator equipped with a shared system-level cache and application-aware management policies, which keeps the programming effort modest. Our approach exploits dataflow information available in the software stack to guide cache replacement (including dead-block prediction), in concert with bypass decisions and mechanisms that alleviate cache thrashing. We assess the proposal using a cycle-accurate simulator and observe substantial performance gains (up to 1.80x speedup) compared with conventional cache architectures. In addition, we build and validate an analytical model that takes into account the actual overlapping behaviors to extend the measurement results of our policies to real-world larger-scale workloads. Experiment results show that when functioning together, our bypassing and thrashing mitigation strategies can handle scenarios both with and without inter-core data sharing and achieve remarkable speedups. Finally, we implement the design in RTL and the area of our design is $\mathbf{0.064mm^2}$ with 15nm process, which can run at 2 GHz clock frequency. Our findings explore the potential of the shared cache design to assist the development of future AI accelerator systems.Open AccessDCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management
The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and their asynchronous management, we investigate the opposite point of the design spectrum: a multi-core AI accelerator equipped with a shared system-level cache and application-aware management policies, which keeps the programming effort modest. Our approach exploits dataflow information available in the software stack to guide cache replacement (including dead-block prediction), in concert with bypass decisions and mechanisms that alleviate cache thrashing. We assess the proposal using a cycle-accurate simulator and observe substantial performance gains (up to 1.80x speedup) compared with conventional cache architectures. In addition, we build and validate an analytical model that takes into account the actual overlapping behaviors to extend the measurement results of our policies to real-world larger-scale workloads. Experiment results show that when functioning together, our bypassing and thrashing mitigation strategies can handle scenarios both with and without inter-core data sharing and achieve remarkable speedups. Finally, we implement the design in RTL and the area of our design is $\mathbf{0.064mm^2}$ with 15nm process, which can run at 2 GHz clock frequency. Our findings explore the potential of the shared cache design to assist the development of future AI accelerator systems. - Open Access
- 8 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 460-469
LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
- 8 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 460-469
Open AccessLLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
- Open Access
- 4 September 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 29 (5) , 1-20
gem5-NVDLA: A Simulation Framework for Compiling, Scheduling, and Architecture Evaluation on AI System-on-Chips
Recent years have seen an increasing trend in designing AI accelerators together with the rest of the system, including CPUs and memory hierarchy. This trend calls for high-quality simulators or analytical models that enable such kind of co-exploration. Currently, the majority of such exploration is supported by AI accelerator analytical models. But such models usually overlook the non-trivial impact of congestion of shared resources, non-ideal hardware utilization and non-zero CPU scheduler overhead, which could only be modeled by cycle-level simulators. However, most simulators with full-stack toolchains are proprietary to corporations, and the few open-source simulators are suffering from either weak compilers or limited space of modeling. This framework resolves these issues by proposing a compilation and simulation flow to run arbitrary Caffe neural network models on the NVIDIA Deep Learning Accelerator (NVDLA) with gem5, a cycle-level simulator, and by adding more building blocks including scratchpad allocation, multi-accelerator scheduling, tensor-level prefetching mechanisms, and a DMA-aided embedded buffer to map workload to multiple NVDLAs. The proposed framework has been tested and verified on a set of convolution neural networks, showcasing the capability of modeling complex buffer management strategies, scheduling policies, and hardware architectures. As a case study of this framework, we demonstrate the importance of adopting different buffering strategies for activation and weight tensors in AI accelerators to acquire remarkable speedup.
- 4 September 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 29 (5) , 1-20
Open Accessgem5-NVDLA: A Simulation Framework for Compiling, Scheduling, and Architecture Evaluation on AI System-on-Chips
Recent years have seen an increasing trend in designing AI accelerators together with the rest of the system, including CPUs and memory hierarchy. This trend calls for high-quality simulators or analytical models that enable such kind of co-exploration. Currently, the majority of such exploration is supported by AI accelerator analytical models. But such models usually overlook the non-trivial impact of congestion of shared resources, non-ideal hardware utilization and non-zero CPU scheduler overhead, which could only be modeled by cycle-level simulators. However, most simulators with full-stack toolchains are proprietary to corporations, and the few open-source simulators are suffering from either weak compilers or limited space of modeling. This framework resolves these issues by proposing a compilation and simulation flow to run arbitrary Caffe neural network models on the NVIDIA Deep Learning Accelerator (NVDLA) with gem5, a cycle-level simulator, and by adding more building blocks including scratchpad allocation, multi-accelerator scheduling, tensor-level prefetching mechanisms, and a DMA-aided embedded buffer to map workload to multiple NVDLAs. The proposed framework has been tested and verified on a set of convolution neural networks, showcasing the capability of modeling complex buffer management strategies, scheduling policies, and hardware architectures. As a case study of this framework, we demonstrate the importance of adopting different buffering strategies for activation and weight tensors in AI accelerators to acquire remarkable speedup.
- Open Access
- 30 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 62-73
LCM: LLM-focused Hybrid SPM-cache Architecture with Cache Management for Multi-Core AI Accelerators
The proliferation of large language models (LLMs) with substantial computational requirements and memory footprints has necessitated the design of more capable AI accelerators. Given the long compilation time of scratchpad memory-based (SPM-based) AI accelerators and the challenges brought by LLMs, we have explored the other side of the tradeoff - a multi-core AI accelerator system that incorporates a shared cache and application-specific management strategies - to provide significantly shorter compilation time at the cost of sometimes slightly lower performance than SPM-based systems. Besides, state-of-the-art mixed precision quantization methods also bring dynamic and irregular memory access patterns that do not fit SPMs well. Experiments conducted on cycle-accurate simulators have shown a maximum 30% memory time reduction with our cache management strategies, and up to 1.37x overall execution speedup with our prefetcher. Furthermore, we integrate the cache model into a state-of-the-art AI accelerator analytical model, allowing for a rapid understanding of the impact of caches for large-scale systems and big ML models. Overall, a system with our managed cache performs comparably with that using a conventional global SPM in non-quantized test-cases. However, our managed cache can outperform an SPM by 50.5% when a mixed precision quantization algorithm is used. With our cache management schemes, the proposed system can achieve a 2.57x speedup compared to LRU caches. Finally, we implement the design in RTL and the area of our design is 0.218mm2 with 15nm process, which can run at 1 GHz clock frequency. Our findings explore the potential of a shared cache design to assist the development of future AI accelerator systems.
- 30 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 62-73
Open AccessLCM: LLM-focused Hybrid SPM-cache Architecture with Cache Management for Multi-Core AI Accelerators
The proliferation of large language models (LLMs) with substantial computational requirements and memory footprints has necessitated the design of more capable AI accelerators. Given the long compilation time of scratchpad memory-based (SPM-based) AI accelerators and the challenges brought by LLMs, we have explored the other side of the tradeoff - a multi-core AI accelerator system that incorporates a shared cache and application-specific management strategies - to provide significantly shorter compilation time at the cost of sometimes slightly lower performance than SPM-based systems. Besides, state-of-the-art mixed precision quantization methods also bring dynamic and irregular memory access patterns that do not fit SPMs well. Experiments conducted on cycle-accurate simulators have shown a maximum 30% memory time reduction with our cache management strategies, and up to 1.37x overall execution speedup with our prefetcher. Furthermore, we integrate the cache model into a state-of-the-art AI accelerator analytical model, allowing for a rapid understanding of the impact of caches for large-scale systems and big ML models. Overall, a system with our managed cache performs comparably with that using a conventional global SPM in non-quantized test-cases. However, our managed cache can outperform an SPM by 50.5% when a mixed precision quantization algorithm is used. With our cache management schemes, the proposed system can achieve a 2.57x speedup compared to LRU caches. Finally, we implement the design in RTL and the area of our design is 0.218mm2 with 15nm process, which can run at 1 GHz clock frequency. Our findings explore the potential of a shared cache design to assist the development of future AI accelerator systems.