Demonstration of transformer-based ALBERT model on a 14nm analog AI inference chip
Open Access
- 30 September 2025
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-11
Abstract
A Lite Bidirectional Encoder Representations from Transformers model is demonstrated on an analog inference chip fabricated at 14nm node with phase change memory. The 7.1 million unique analog weights shared across 12 layers are mapped to a single chip, accurately programmed into the conductance of 28.3 million devices, for this first analog hardware demonstration of a meaningfully large Transformer model. The implemented model achieved near iso-accuracy on the General Language Understanding Evaluation benchmark of seven tasks, despite the presence of weight-programming errors, hardware imperfections, readout noise, and error propagation. The average hardware accuracy was only 1.8% below that of the floating-point reference, with several tasks at full iso-accuracy. Careful fine-tuning of model weights using hardware-aware techniques contributes an average hardware accuracy improvement of 4.4%. Accuracy loss due to conductance drift – measured to be roughly 5% over 30 days – was reduced to less than 1% with a recalibration-based “drift compensation” technique.Keywords
This publication has 38 references indexed in Scilit:
- Electrochemical random-access memory: recent advances in materials, devices, and systems towards neuromorphic computingNano Convergence, 2024
- Programming memristor arrays with arbitrarily high precision for analog computingScience, 2024
- Novel nanocomposite-superlattices for low energy and high stability nanoscale phase-change memoryNature Communications, 2024
- First demonstration of in-memory computing crossbar using multi-level Cell FeFETNature Communications, 2023
- Open-loop analog programmable electrochemical memory arrayNature Communications, 2023
- Edge learning using a fully integrated neuro-inspired memristor chipScience, 2023
- Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorsNature Communications, 2023
- An analog-AI chip for energy-efficient speech recognition and transcriptionNature, 2023
- A compute-in-memory chip based on resistive random-access memoryNature, 2022
- A crossbar array of magnetoresistive memory devices for in-memory computingNature, 2022