ClipFormer : Key–Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation

Authors:
Publisher website
Google Scholar
Abstract
Transformers have revolutionized various real-world applications from natural language processing to computer vision. However, traditional von-Neumann computing paradigm faces memory and bandwidth limitations in accelerating transformers owing to their massive model sizes. To this end, in-memory computing (IMC) crossbars based on nonvolatile memories (NVMs), due to their ability to perform highly parallelized matrix-vector multiplications (MVMs) with high energy efficiencies, have emerged as a promising solution for accelerating transformers. However, analog MVM operations in crossbars introduce nonidealities, such as stochastic read and write noise, which affect the inference accuracy of the deployed transformers. Specifically, we find pretrained vision transformers (ViTs) to be vulnerable on crossbars due to the impact of write noise on the dynamically generated key $(K)$ and value $(V)$ matrices in the attention layers, an effect not accounted for in prior studies. We, thus, propose ClipFormer, a transformation on the K and V matrices during inference, to boost the nonideal accuracies of pretrained ViT models. ClipFormer requires no additional hardware and training overhead and is amenable to transformers deployed on any memristive crossbar platform impacted by write nonidealities. Our experiments on Imagenet-1k dataset using pretrained DeiT-S transformers, subjected to standard training and variation-aware training, show >10%–40% higher nonideal accuracies at the high write noise regime by applying ClipFormer.
Funding Information
  • CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC
  • National Science Foundation (2312366, 2318152)
  • TII
  • DoE MMICC center SEA-CROGS (DE-SC0023198)

This publication has 33 references indexed in Scilit: