AGO: Boosting Mobile AI Inference Performance by Removing Constraints on Graph Optimization
Authors:
- 17 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-10
Abstract
Traditional deep learning compilers rely on heuristics for subgraph generation, which impose extra constraints on graph optimization, e.g., each subgraph can only contain at most one complex operator. In this paper, we propose AGO, a framework for graph optimization with arbitrary structures to boost the inference performance of deep models by removing such constraints. To create new optimization opportunities for complicated subgraphs, we propose intensive operator fusion, which effectively stitches multiple complex operators together for better performance. Further, we design a graph partitioning scheme that allows an arbitrary structure for each subgraph while guaranteeing the acyclic property among all generated subgraphs. Additionally, to enable efficient performance tuning for complicated subgraphs, we devise a divide-and-conquer tuning mechanism to orchestrate different system components. Through extensive experiments on various neural networks and mobile devices, we show that our system can improve the inference performance by up to 3.3× when compared with state-of-the-art vendor libraries and deep compilers.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
This publication has 13 references indexed in Scilit:
- Analytical characterization and design space exploration for optimization of CNNs2021
- CLIO2020
- SPINN2020
- FlexTensor2020
- TASO2019
- MnasNet: Platform-Aware Neural Architecture Search for Mobile2019
- Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code2019
- Relay: a new IR for machine learning frameworks2018
- Pregel2010
- ImageNet: A large-scale hierarchical image database2009