Randomized matrix sketching for neural network training and gradient monitoring
- 11 April 2026
- journal article
- research article
- Published by World Scientific Pub Co Pte Ltd in Mathematical Models and Methods in Applied Sciences
- Vol. 36 (08) , 1623-1648
Abstract
Neural network training relies on gradient computation through backpropagation, yet memory requirements for storing layer activations present significant scalability challenges. We present the first adaptation of control-theoretic matrix sketching to neural network layer activations, enabling memory-efficient gradient reconstruction in backpropagation. This work builds on recent matrix sketching frameworks for dynamic optimization problems, where similar state trajectory storage challenges motivate sketching techniques. Our approach sketches layer activations using three complementary sketch matrices maintained through exponential moving averages with adaptive rank adjustment, automatically balancing memory efficiency against approximation quality. Empirical evaluation on MNIST, CIFAR-10, and physics-informed neural networks demonstrates a controllable accuracy-memory tradeoff. We demonstrate a gradient monitoring application on MNIST showing how sketched activations enable real-time gradient norm tracking with minimal memory overhead. These results establish that sketched activation storage provides a viable path toward memory-efficient neural network training and analysis.Keywords
All Related Versions
Funding Information
- Office of Naval Research (Award No. N00014-24-1-2147)
- National Sleep Foundation (DMS-2408877)
- Air Force Office of Scientific Research (Award No. FA9550-25-1-0231)
This publication has 15 references indexed in Scilit:
- An inexact semismooth Newton method with application to adaptive randomized sketching for dynamic optimizationFinite Elements in Analysis and Design, 2023
- Physics-informed machine learningNature Reviews Physics, 2021
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equationsJournal of Computational Physics, 2019
- Streaming Low-Rank Matrix Approximation with an Application to Scientific SimulationSIAM Journal on Scientific Computing, 2019
- Practical Sketching Algorithms for Low-Rank Matrix ApproximationSIAM Journal on Matrix Analysis and Applications, 2017
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- A review of the adjoint-state method for computing the gradient of a functional with geophysical applicationsGeophysical Journal International, 2006
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992
- Learning representations by back-propagating errorsNature, 1986