Leland Chang
Research Subjects
Leland Chang has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Swagath Venkataramani
IBM T.J. Watson Research Center Yorktown Heights New York USA
2 shared publications Anantha P. Chandrakasan
Microsystems Technology Laboratories, Massachusetts Institute of Technology, Cambridge, MA, USA
1 shared publication- Andrea Fasoli
IBM Research USA
1 shared publication - Chi-Chun Liu
IBM Research USA
1 shared publication - Eri Ogawa
IBM Research Labs
1 shared publication - Hiroshi Inoue
IBM Research -Tokyo
1 shared publication
Latest Publications
- Open Access
Is Finer Better? The Limits of Microscaling Formats in Large Language Models
Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.
Open AccessIs Finer Better? The Limits of Microscaling Formats in Large Language Models
Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.
- 31 July 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 39 (5) , 102-111
DeepTools: Compiler and Execution Runtime Extensions for RaPiD AI Accelerator
Systems specialized for Artificial Intelligence (AI) workloads are increasingly deployed across the spectrum of computing devices from edge to the cloud. However, their ubiquitous adoption requires bridging two seemingly conflicting challenges - the need to deliver extreme processing efficiencies while employing intuitive (and familiar) programming and user interfaces, making AI systems compelling even for common (non-expert) users. In this paper, we take a significant first step towards this goal and present an end-to-end software stack for the RAPID AI accelerator developed by IBM Research. We achieve this by developing a set of software extensions, called DEEPTOOLS, that leverage and work within popular deep learning frameworks (e.g., TensorFlow), thus requiring no additional user input while still enabling aggressive, accelerator-specific performance optimization that might otherwise be expected from a full, custom framework. DEEPTOOLS has two key components: (i) a compiler runtime called DEEPRT, which given a DNN graph automatically identifies how best to execute it on RAPID and constructs the requisite program binaries, (ii) an execution runtime called RAPIDLIB, which triggers and manages the execution of compute and data-transfer operations on RAPID. We demonstrate the utility of DEEPTOOLS by plugging it into TensorFlow and mapping multiple large and practical DNNs (AlexNet, VGG, ResNet, and LSTM) to RAPID. Further, our unique optimizations yielded substantial improvement in performance over hand-tuned homogeneous mappings.
- 31 July 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Micro
- Vol. 39 (5) , 102-111
DeepTools: Compiler and Execution Runtime Extensions for RaPiD AI Accelerator
Systems specialized for Artificial Intelligence (AI) workloads are increasingly deployed across the spectrum of computing devices from edge to the cloud. However, their ubiquitous adoption requires bridging two seemingly conflicting challenges - the need to deliver extreme processing efficiencies while employing intuitive (and familiar) programming and user interfaces, making AI systems compelling even for common (non-expert) users. In this paper, we take a significant first step towards this goal and present an end-to-end software stack for the RAPID AI accelerator developed by IBM Research. We achieve this by developing a set of software extensions, called DEEPTOOLS, that leverage and work within popular deep learning frameworks (e.g., TensorFlow), thus requiring no additional user input while still enabling aggressive, accelerator-specific performance optimization that might otherwise be expected from a full, custom framework. DEEPTOOLS has two key components: (i) a compiler runtime called DEEPRT, which given a DNN graph automatically identifies how best to execute it on RAPID and constructs the requisite program binaries, (ii) an execution runtime called RAPIDLIB, which triggers and manages the execution of compute and data-transfer operations on RAPID. We demonstrate the utility of DEEPTOOLS by plugging it into TensorFlow and mapping multiple large and practical DNNs (AlexNet, VGG, ResNet, and LSTM) to RAPID. Further, our unique optimizations yielded substantial improvement in performance over hand-tuned homogeneous mappings.
- 3 December 2010
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 46 (1) , 85-96
A 512kb 8T SRAM Macro Operating Down to 0.57 V With an AC-Coupled Sense Amplifier and Embedded Data-Retention-Voltage Sensor in 45 nm SOI CMOS
An 8T SRAM fabricated in 45 nm SOI CMOS exhibits voltage scalable operation from 1.2V down to 0.57V with access times from 400 ps to 3.4 ns. Timing variation and the challenge of low voltage operation are addressed with an AC-coupled sense amplifier. An area efficient data path is achieved with a regenerative global bitline scheme. Finally, a data retention voltage sensor has been developed to predict the mismatch-limited minimum standby voltage without corrupting the contents of the memory.
- 3 December 2010
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Solid-State Circuits
- Vol. 46 (1) , 85-96
A 512kb 8T SRAM Macro Operating Down to 0.57 V With an AC-Coupled Sense Amplifier and Embedded Data-Retention-Voltage Sensor in 45 nm SOI CMOS
An 8T SRAM fabricated in 45 nm SOI CMOS exhibits voltage scalable operation from 1.2V down to 0.57V with access times from 400 ps to 3.4 ns. Timing variation and the challenge of low voltage operation are addressed with an AC-coupled sense amplifier. An area efficient data path is achieved with a regenerative global bitline scheme. Finally, a data retention voltage sensor has been developed to predict the mismatch-limited minimum standby voltage without corrupting the contents of the memory.