Denseflex: A Low Rank Factorization Methodology for Adaptable Dense Layers in DNNs
Open Access
- 7 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 21-31
Abstract
Low-Rank Factorization (LRF) is a popular compression technique used in Deep Neural Networks (DNNs). LRF can reduce both the memory size and the arithmetic operations in a DNN layer by approximating a weight tensor/matrix by two or more smaller tensors/matrices. Employing LRF to DNN is a challenging task for several reasons. First, the exploration space is massive and different solutions provide different trade-offs among memory, FLOPs, inference time, and validation accuracy; second, multiple DNN layers and multiple LRF algorithms must be considered; third, every extracted solution must undergo through a calibration phase and this makes the LRF process time-consuming. In this paper, a methodology, called Denseflex, is presented that formulates the LRF problem as an inference time vs. FLOPs vs. memory vs. validation accuracy Design Space Exploration (DSE) problem. Moreover, to the best of our knowledge, this is the first work that proposes a methodology to efficiently combine two different LRF methods (Singular Value Decomposition -SVD- and Tensor Train Decomposition -TTD-) in the same framework. Denseflex is formulated as a design tool in which the user can provide specific memory, FLOPs, and/or execution time constraints and the tool will output a set of solutions that meet the given constraints avoiding the time-consuming re-training phases. Our results indicate that our approach is able to prune the design space by 62% (on average) over related works for nine DNN models (up to 88% in AlexNet), while the extracted LRF solutions exhibit both lower memory footprints and lower execution times compared to the initial model.Keywords
This publication has 28 references indexed in Scilit:
- Towards Energy Efficient DNN accelerator via Sparsified Gradual Knowledge Distillation2022
- A Design Space Exploration Methodology for Enabling Tensor Train Decomposition in Edge Devices2022
- Explainable AI decision model for ECG data of cardiac disordersBiomedical Signal Processing and Control, 2022
- Two Might Do: A Beat-by-Beat Classification of Cardiac Abnormalities Using Deep Learning with Domain-Specific Features2021
- Pruning from ScratchProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Information theory based pruning for CNN compression and its application to image classification and action recognition2019
- Constrained Optimization Based Low-Rank Approximation of Deep Neural Networks2018
- Singular Value Decomposition2010
- Valgrind2007
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998