TMAC: Training-Targeted Mapping and Architecture Co-Exploration for Wafer-Scale Chips
Open Access
- 1 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Integrated Circuits and Systems
- Vol. 1 (4) , 178-195
Abstract
Transformer-based large language models (LLMs) have made significant strides in the field of artificial intelligence (AI). However, training these LLMs imposes immense demands on computational power and bandwidth for hardware systems. Wafer-scale chips (WSCs) offer a promising solution, yet they struggle with limited on-chip memory and complex tensor partitioning. To fully harness the high-bandwidth, low-latency on-chip interconnect benefits of WSCs and to alleviate the on-chip memory limitations, a specialized mapping and architecture co-exploration method is essential. Despite existing efforts in memory optimization and mapping, current approaches fall short for WSC scenarios. To bridge this gap, we introduce TMAC, an architecture-mapping co-exploration framework that integrates recomputation into the design space, fully exploiting optimization opportunities overlooked by existing works. Further, TMAC takes advantage of the superior on-chip interconnect performance of WSCs by incorporating a more flexible tensor partition scheme. TMAC then introduces a novel operator-centric encoding scheme (OCES) designed to comprehensively describe the mapping space for training LLMs. Unlike previous studies that focus solely on communication volume analysis based on mapping, TMAC explores the design space by evaluating the combined impact of mapping and architecture on training performance. However, fully accounting for these untapped optimization opportunities increases the complexity of the design space. To address this, we streamline the simulation process, reducing the time needed for exploration. Compared to AccPar, Deepspeed and Megatron, TMAC delivers a $3.1\times$ , $2.9\times$ , $1.6\times$ performance gain. In terms of memory usage, TMAC requires $3.6\times$ , $3.1\times$ less memory than AccPar and Deepspeed, respectively and is comparable to Megatron's full recomputation method.Keywords
Funding Information
- National Science and Technology (2022ZD0115200)
- Frontier Technique Collaboration (QYJS-2023-2801-B)
- National Natural Science Foundation of China (62125403, 92164301)
- Beijing S&T (Z221100007722023)
- Shanghai Municipal Science and Technology
- 2022 Special Project on Industrial Foundation Reconstruction and High Quality Development of Manufacturing Industry (CEIEC-2022-ZM02-0245)
- Beijing National Research Center For Information Science And Technology
- Beijing Advanced Innovation Center for Integrated Circuits
This publication has 75 references indexed in Scilit:
- Generative AIBusiness & Information Systems Engineering, 2023
- Protrusion of Through-Silicon-Via (TSV) Copper with Double Annealing ProcessesJournal of Electronic Materials, 2022
- A review on the mainstream through-silicon via etching methodsMaterials Science in Semiconductor Processing, 2022
- A congestion-aware routing algorithm for mesh-based platform networks-on-chipMicroelectronics Journal, 2021
- ERDSE: efficient reinforcement learning based design space exploration method for CNN accelerator on resource limited platformGraphics and Visual Computing, 2021
- Gradient-based optimizer: A new metaheuristic optimization algorithmInformation Sciences, 2020
- Differential evolution algorithm with multiple mutation strategies based on roulette wheel selectionApplied Intelligence, 2018
- Roulette-wheel selection via stochastic acceptancePhysica A: Statistical Mechanics and its Applications, 2012
- Congestion-aware channel assignment for multi-channel wireless mesh networksComputer Networks, 2009
- The communication challenge for MPP: Intel Paragon and Meiko CS-2Parallel Computing, 1994