Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention
Open Access
- 2 April 2024
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 13 (7) , 1330
Abstract
Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network’s performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.Keywords
Funding Information
- National Natural Science Foundation of China (U23A20343, 11971374, 61821005, 61903358, YSBR-041, Y202051, 2022196)
- CAS Project for Young Scientists in Basic Research (U23A20343, 11971374, 61821005, 61903358, YSBR-041, Y202051, 2022196)
- Youth Innovation Promotion Association of the Chinese Academy of Sciences (U23A20343, 11971374, 61821005, 61903358, YSBR-041, Y202051, 2022196)
This publication has 36 references indexed in Scilit:
- ResU-Former: Advancing Remote Sensing Image Segmentation with Swin Residual Transformer for Precise Global–Local Feature Recognition and Visual–Semantic Space LearningElectronics, 2024
- HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural NetworksProceedings of the AAAI Conference on Artificial Intelligence, 2023
- A novel compact design of convolutional layers with spatial transformation towards lower-rank representation for image classificationKnowledge-Based Systems, 2022
- Deep neural network compression by Tucker decomposition with nonlinear responseKnowledge-Based Systems, 2022
- Achieving adversarial robustness via sparsityMachine Learning, 2021
- Deep Neural Network Compression by In-Parallel Pruning-QuantizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network scienceNature Communications, 2018
- Accelerating Very Deep Convolutional Networks for Classification and DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Tensor-Train DecompositionSIAM Journal on Scientific Computing, 2011
- Some Mathematical Notes on Three-Mode Factor AnalysisPsychometrika, 1966