Runtime Performance Prediction for Deep Learning Models with Graph Neural Network
Authors:
- 1 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 368-380
Abstract
Deep learning models have been widely adopted in many application domains. Predicting the runtime performance of deep learning models, such as GPU memory consumption and training time, is important for boosting development productivity and reducing resource waste. The reason is that improper configurations of hyperparameters and neural architectures can result in many failed training jobs or unsatisfactory models. However, the runtime performance prediction of deep learning models is challenging because of the hybrid programming paradigm, complicated hidden factors within the framework runtime, enormous model configuration space, and broad differences among models. In this paper, we propose DNNPerf, a novel ML-based tool for predicting the runtime performance of deep learning models using Graph Neural Network. DNNPerf represents a model as a directed acyclic computation graph and incorporates a rich set of performance-related features based on the computational semantics of both nodes and edges. We also propose a new Attention-based Node-Edge Encoder for the node and edge features. DNNPerf is evaluated on thousands of configurations of real-world and synthetic deep learning models to predict their GPU memory consumption and training time. The experimental results show that DNNPerf achieves accurate predictions, with an overall error of 7.4% for the training time prediction and an overall error of 13.7% for the GPU memory consumption prediction, confirming its effectiveness.Keywords
This publication has 34 references indexed in Scilit:
- Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T)2015
- Performance Prediction of Configurable Software Systems by Fourier Learning (T)2015
- Performance-influence models for highly configurable systems2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Variability-aware performance prediction: A statistical learning approach2013
- Scalable Prediction of Non-functional Properties in Software Product Lines2011
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Long Short-Term MemoryNeural Computation, 1997
- Learning representations by back-propagating errorsNature, 1986
- Neural networks and physical systems with emergent collective computational abilities.Proceedings of the National Academy of Sciences of the United States of America, 1982