A Hardware Design Framework for Computer Vision Models Based on Reconfigurable Devices
Open Access
- 15 January 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Reconfigurable Technology and Systems
- Vol. 17 (1) , 1-31
Abstract
In computer vision, the joint development of the algorithm and computing dimensions cannot be separated. Models and algorithms are constantly evolving, while hardware designs must adapt to new or updated algorithms. Reconfigurable devices are recognized as important platforms for computer vision applications because of their reconfigurability. There are two typical design approaches: customized and overlay design. However, existing work is unable to achieve both efficient performance and scalability to adapt to a wide range of models. To address both considerations, we propose a design framework based on reconfigurable devices to provide unified support for computer vision models. It provides software-programmable modules while leaving unit design space for problem-specific algorithms. Based on the proposed framework, we design a model mapping method and a hardware architecture with two processor arrays to enable dynamic and static reconfiguration, thereby relieving redesign pressure. In addition, resource consumption and efficiency can be balanced by adjusting the hyperparameter. In experiments on CNN, vision Transformer, and vision MLP models, our work’s throughput is improved by 18.8x–33.6x and 1.4x–2.0x compared to CPU and GPU. Compared to others on the same platform, accelerators based on our framework can better balance resource consumption and efficiency.Keywords
Funding Information
- National Natural Science Foundation of China (61972293)
This publication has 17 references indexed in Scilit:
- Synetgy2019
- Hardware/Software Codesign for Convolutional Neural Networks Exploiting Dynamic Partial Reconfiguration on PYNQ2018
- Serving DNNs in Real Time at Datacenter Scale with Project BrainwaveIEEE Micro, 2018
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- FINN2017
- Caffeinated FPGAs: FPGA framework For Convolutional Neural Networks2016
- Efficient hardware architecture of softmax layer in deep neural network2016
- Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks2015
- FPGA-implementation of high-speed MLP neural network2011
- ImageNet: A large-scale hierarchical image database2009