DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management
Preprint
Abstract
The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and their asynchronous management, we investigate the opposite point of the design spectrum: a multi-core AI accelerator equipped with a shared system-level cache and application-aware management policies, which keeps the programming effort modest. Our approach exploits dataflow information available in the software stack to guide cache replacement (including dead-block prediction), in concert with bypass decisions and mechanisms that alleviate cache thrashing. We assess the proposal using a cycle-accurate simulator and observe substantial performance gains (up to 1.80x speedup) compared with conventional cache architectures. In addition, we build and validate an analytical model that takes into account the actual overlapping behaviors to extend the measurement results of our policies to real-world larger-scale workloads. Experiment results show that when functioning together, our bypassing and thrashing mitigation strategies can handle scenarios both with and without inter-core data sharing and achieve remarkable speedups. Finally, we implement the design in RTL and the area of our design is $\mathbf{0.064mm^2}$ with 15nm process, which can run at 2 GHz clock frequency. Our findings explore the potential of the shared cache design to assist the development of future AI accelerator systems.
Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- TETRISACM SIGARCH Computer Architecture News, 2017
- Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks2016
- Analytical Processor Performance and Power Modeling using Micro-Architecture Independent CharacteristicsIEEE Transactions on Computers, 2016
- Open Cell Library in 15nm FreePDK Technology2015
- Exploring Heterogeneous NoC Design Space in Heterogeneous GPU-CPU ArchitecturesJournal of Computer Science and Technology, 2015
- Chisel2012
- SHiP2011
- Counter-Based Cache Replacement and Bypassing AlgorithmsIEEE Transactions on Computers, 2008
- IATAC: a smart predictor to turn-off L2 cache linesACM Transactions on Architecture and Code Optimization, 2005
- Memory coherence activity prediction in commercial workloads2004