A Heterogeneous CNN Compilation Framework for RISC-V CPU and NPU Integration Based on ONNX-MLIR
- 25 April 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 44 (11) , 4222-4235
Abstract
With the continuous advancement of convolutional neural networks (CNNs), many neural network processing units (NPUs) have emerged in recent years. NPUs offer advantages, such as improved energy efficiency and low latency compared to traditional processors. However, NPUs often struggle to keep pace with the growing complexity of algorithmic models, which limits the implementation of certain AI applications. Central processing units (CPUs) are known for their versatility, but are computationally inefficient. Combining the strengths of both architectures could present a viable solution to these challenges. Despite this potential, there is currently insufficient academic focus on CPU/NPU co-computing, and few architectures or toolchains have been developed to support such integration. In this article, we propose a RISC-V CPU/NPU co-computing-based compilation framework to solve the problem of mapping from algorithms to heterogeneous architectures. To bridge the computational gap between heterogeneous platforms, within ONNX-MLIR, we developed three phases to support operator packing, algorithm adaption, and data coordination. To integrate heterogeneous compilation platforms, we introduce four additional functions to handle task scheduling, weight quantization, weight rearrangement, and memory management. In particular: 1) to optimize the transition overhead, an operator packing technique is introduced by extending the ONNX-MLIR framework with custom operators, which enhances NPU efficiency by reducing the number of transitions between operations; 2) at the aspect of task scheduling, a three-mode task scheduler is developed to allow users to customize task distribution for correctness verification, performance optimization, or detailed analysis on heterogeneous platforms, which demonstrates the versatility of our framework; and 3) In terms of memory management, an optimized scheme for CPU/NPU architectures is proposed to ensure accurate data access. This memory management employs a dual-end approach with self-checks and a release-after-use mechanism to further improve efficiency. Experimental results demonstrate that the framework effectively generates instructions for various CNNs, significantly enhancing the efficiency and performance of heterogeneous architectures. It achieves up to a $7.06\times $ reduction in code density and up to a $5.58\times $ improvement in memory usage, narrowing the computational gap between CPUs and NPUs.
Keywords
Funding Information
- National Natural Science Foundation of China (62271300)
- National Natural Science Foundation of China (12141107)
- Shanghai Science and Technology Plan Project (24DP1501100)
- Shanghai Science and Technology Plan Project (24DP1500600)
This publication has 25 references indexed in Scilit:
- Designing RISC-V Instruction Set Extensions for Artificial Neural Networks: An LLVM Compiler-Driven PerspectiveIEEE Access, 2024
- A General-Purpose Compiler Design for Instruction-Based AI Accelerator Implementation2023
- Loop-Tiling Based Compiling Optimization for CNN Accelerators2023
- Accelerating Neural Networks Using Open Standard Software on RISC-V2023
- Configurable CNN Accelerator in Speech Processing based on Vector Convolution2022
- A CNN Hardware Accelerator Designed for YOLO Algorithm Based on RISC-V SoC2021
- Customized Instruction on RISC-V for Winograd-Based Convolution Acceleration2021
- Data Parallel C++2021
- RV-CNN: Flexible and Efficient Instruction Set for CNNs Based on RISC-V Processors2019
- DeepFace: Closing the Gap to Human-Level Performance in Face Verification2014