Principal Component Networks: Utilizing Low-Rank Activation Structure to Reduce Parameters Early in Training
Open Access
- 8 April 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM / IMS Journal of Data Science
- Vol. 1 (2) , 1-27
Abstract
Recent works show that overparameterized neural networks contain small subnetworks that exhibit comparable accuracy to the full model when trained in isolation. These results highlight the potential to reduce the computational costs of deep neural network training without sacrificing generalization performance. Initial approaches for finding these small networks relied on expensive multi-round train-and-prune procedures, limiting their practical potential, but more recent work identifies subnetworks using structured pruning techniques early in training. In this article, we study network activations, rather than network weights, and find that hidden layer activations in overparameterized networks exist primarily in subspaces smaller than the actual model width. We further notice that these subspaces can be identified early in training. Based on these observations, we show how to efficiently find small networks that exhibit similar accuracy to their overparameterized counterparts after only a few training epochs. We term these network architectures Principal Component Networks (PCNs). PCNs compress individual layers by retaining only the high variance linear combinations of channels—defined by the principal components of the layer inputs—a key difference from structured pruning techniques which focus on individual channel pruning using localized channel measurements. We evaluate PCNs on CIFAR-10 and ImageNet for VGG and ResNet style architectures and compare against existing methods for subnetwork identification during early training. We find that PCNs consistently reduce parameter counts with little accuracy loss, thus providing the potential to reduce the computational costs of deep neural network training. Beyond model compression, we also connect our observation regarding hidden layer activations to the feature representations learned by neural networks and discuss areas for future work.Keywords
Funding Information
- Defense Advanced Research Projects Agency (ASKE HR00111990013 and ASKEM HR001122S0005)
This publication has 18 references indexed in Scilit:
- What’s Hidden in a Randomly Weighted Neural Network?2020
- Bag of Tricks for Image Classification with Convolutional Neural Networks2019
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- Training Sparse Neural Networks2017
- Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning2017
- Deep Residual Learning for Image Recognition2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Efficient and accurate approximations of nonlinear convolutional networks2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998