LCLeland Chang

Leland Chang

*This is an algorithmically generated author record based on publicly available information.

Leland Chang has a total of 20 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Is Finer Better? The Limits of Microscaling Formats in Large Language Models

    by Andrea Fasoli, Monodeep Kar, Chi-Chun Liu, Swagath Venkataramani, Viji Srinivasan, Leland Chang, Naigang Wang

    Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.

  • DeepTools: Compiler and Execution Runtime Extensions for RaPiD AI Accelerator

    by Swagath Venkataramani, Jungwook Choi, Vijayalakshmi Srinivasan, Wei Wang, Jintao Zhang, Marcel Schaal, Mauricio J. Serrano, Kazuaki Ishizaki, Hiroshi Inoue, Eri Ogawa, et al.

    Systems specialized for Artificial Intelligence (AI) workloads are increasingly deployed across the spectrum of computing devices from edge to the cloud. However, their ubiquitous adoption requires bridging two seemingly conflicting challenges - the need to deliver extreme processing efficiencies while employing intuitive (and familiar) programming and user interfaces, making AI systems compelling even for common (non-expert) users. In this paper, we take a significant first step towards this goal and present an end-to-end software stack for the RAPID AI accelerator developed by IBM Research. We achieve this by developing a set of software extensions, called DEEPTOOLS, that leverage and work within popular deep learning frameworks (e.g., TensorFlow), thus requiring no additional user input while still enabling aggressive, accelerator-specific performance optimization that might otherwise be expected from a full, custom framework. DEEPTOOLS has two key components: (i) a compiler runtime called DEEPRT, which given a DNN graph automatically identifies how best to execute it on RAPID and constructs the requisite program binaries, (ii) an execution runtime called RAPIDLIB, which triggers and manages the execution of compute and data-transfer operations on RAPID. We demonstrate the utility of DEEPTOOLS by plugging it into TensorFlow and mapping multiple large and practical DNNs (AlexNet, VGG, ResNet, and LSTM) to RAPID. Further, our unique optimizations yielded substantial improvement in performance over hand-tuned homogeneous mappings.


  • A 512kb 8T SRAM Macro Operating Down to 0.57 V With an AC-Coupled Sense Amplifier and Embedded Data-Retention-Voltage Sensor in 45 nm SOI CMOS

    by Masood Qazi, Kevin Stawiasz, Leland Chang, Anantha P. Chandrakasan

    An 8T SRAM fabricated in 45 nm SOI CMOS exhibits voltage scalable operation from 1.2V down to 0.57V with access times from 400 ps to 3.4 ns. Timing variation and the challenge of low voltage operation are addressed with an AC-coupled sense amplifier. An area efficient data path is achieved with a regenerative global bitline scheme. Finally, a data retention voltage sensor has been developed to predict the mismatch-limited minimum standby voltage without corrupting the contents of the memory.