On the Convergence Theory of Pipeline Gradient-Based Analog In-Memory Training
- 29 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Selected Areas in Information Theory
- Vol. 7, 279-289
Abstract
Aiming to accelerate the training of large deep neural networks (DNN) in an energy-efficient way, analog inmemory computing (AIMC) emerges as a solution with immense potential. AIMC accelerator keeps model weights in memory without moving them from memory to processors during training, reducing overhead dramatically. Despite its efficiency, scaling up AIMC systems presents significant challenges. Since weight copying is expensive and inaccurate, data parallelism is less efficient on AIMC accelerators. It necessitates the exploration of pipeline parallelism, particularly asynchronous pipeline parallelism, which utilizes all available accelerators during the training process. This paper examines the convergence theory of stochastic gradient descent on AIMC hardware with an asynchronous pipeline (Analog-SGD-AP). Although there is empirical exploration of AIMC accelerators, the theoretical understanding of how analog hardware imperfections in weight updates affect the training of multi-layer DNN models remains underexplored. Furthermore, the asynchronous pipeline parallelism results in stale weights issues, which render the update signals no longer valid gradients. To close the gap, this paper investigates the convergence properties of Analog-SGD-AP on multi-layer DNN training. We show that the Analog-SGD-AP converges with iteration complexity O (ε–2+ε–1) despite the aforementioned issues, which matches the complexities of digital SGD and Analog SGD with synchronous pipeline, except the non-dominant term O(ε–1). It implies that AIMC training benefits from asynchronous pipelining almost for free compared with the synchronous pipeline by overlapping computation.Keywords
All Related Versions
Funding Information
- International Business Machines (IBM) through the IBM-Rensselaer Future of Computing Research Collaboration
- National Science Foundation (2401297)
- National Science Foundation (2532349)
- National Science Foundation (2532653)
- Cisco Research Award
This publication has 26 references indexed in Scilit:
- On the Convergence Theory of Pipeline Gradient-Based Analog In-Memory TrainingIEEE Journal on Selected Areas in Information Theory, 2026
- Fast and robust analog in-memory deep neural network trainingNature Communications, 2024
- Loss landscapes and optimization in over-parameterized non-linear systems and neural networksApplied and Computational Harmonic Analysis, 2022
- A Flexible and Fast PyTorch Toolkit for Simulating Training and Inference on Analog Crossbar Arrays2021
- On the Acceleration of Deep Learning Model Parallelism With Staleness2020
- In situ training of feed-forward and recurrent convolutional memristor networksNature Machine Intelligence, 2019
- Wider or Deeper: Revisiting the ResNet Model for Visual RecognitionPattern Recognition, 2019
- Face classification using electronic synapsesNature Communications, 2017
- PipeLayer: A Pipelined ReRAM-Based Accelerator for Deep Learning2017
- Experimental Demonstration and Tolerancing of a Large-Scale Neural Network (165 000 Synapses) Using Phase-Change Memory as the Synaptic Weight ElementIEEE Transactions on Electron Devices, 2015