Fused Depthwise Tiling for Memory Optimization in TinyML Deep Neural Network Inference
Preprint
Abstract
Memory optimization for deep neural network (DNN) inference gains high relevance with the emergence of TinyML, which refers to the deployment of DNN inference tasks on tiny, low-power microcontrollers. Applications such as audio keyword detection or radar-based gesture recognition are heavily constrained by the limited memory on such tiny devices because DNN inference requires large intermediate run-time buffers to store activations and other intermediate data, which leads to high memory usage. In this paper, we propose a new Fused Depthwise Tiling (FDT) method for the memory optimization of DNNs, which, compared to existing tiling methods, reduces memory usage without inducing any run time overhead. FDT applies to a larger variety of network layers than existing tiling methods that focus on convolutions. It improves TinyML memory optimization significantly by reducing memory of models where this was not possible before and additionally providing alternative design points for models that show high run time overhead with existing methods. In order to identify the best tiling configuration, an end-to-end flow with a new path discovery method is proposed, which applies FDT and existing tiling methods in a fully automated way, including the scheduling of the operations and planning of the layout of buffers in memory. Out of seven evaluated models, FDT achieved significant memory reduction for two models by 76.2% and 18.1% where existing tiling methods could not be applied. Two other models showed a significant run time overhead with existing methods and FDT provided alternative design points with no overhead but reduced memory savings.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- DeFiNES: Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators through Analytical Modeling2023
- Memory-Throughput Trade-off for CNN-Based Applications at the EdgeACM Transactions on Design Automation of Electronic Systems, 2022
- A Survey of Quantization Methods for Efficient Neural Network Inference2022
- DeeperThings: Fully Distributed CNN Inference on Resource-Constrained Edge DevicesInternational Journal of Parallel Programming, 2021
- DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUsIEEE Transactions on Computers, 2021
- ZigZag: Enlarging Joint Architecture-Mapping Design Space Exploration for DNN AcceleratorsIEEE Transactions on Computers, 2021
- High-Utilization, High-Flexibility Depth-First CNN Coprocessor for Image Pixel Processing on FPGAIEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2021
- DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge ClustersIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2018
- Recent progress in analog memory-based accelerators for deep learningJournal of Physics D: Applied Physics, 2018
- Scheduling series-parallel task graphs to minimize peak memoryTheoretical Computer Science, 2018