Don't Buy the Pig in a Poke: Benchmarking DNNs Inference Performance before Development
Open Access
- 1 January 2024
- conference paper
- Published by HICSS Conference Office
Abstract
As deep neural networks (DNNs) are increasingly used in practical settings, developers often adopt pre-existing DNN architectures from a vast collection of well-established models. However, in industrial environments, factors beyond simply achieving high accuracy are becoming important. The runtime performance is critical, as delays can lead to conveyor stops or equipment damage. Currently, determining the runtime performance of a DNN requires multiple iterations and testing specific configurations, whereas existing methods for benchmarking DNNs mainly compare different hardware or runtime parameters. We present tritonPerf, an approach to obtain and compare the runtime (latency and throughput) for a broad range of settings and existing DNN architectures in the final application environment. It allows data scientists to evaluate the performance and compare the results for a wide range of models before time and resource-intensive hyperparameter tuning is performed. We demonstrate the gain of tritonPerf in an extensive field study using an industrial setting, where we benchmark and compare the runtime of 57 different models.Keywords
This publication has 9 references indexed in Scilit:
- LiveDewStream: A stream processing platform for running in-lab distributed deep learning inferences on smartphone clusters at the edgeSoftwareX, 2022
- A Deep Learning Framework Performance Evaluation to Use YOLO in Nvidia Jetson PlatformApplied Sciences, 2022
- A DNN Inference Latency-aware GPU Power Management Scheme2021
- TensorFlow2021
- Benchmarking AI Inference: Where we are in 20202021
- Pre-trained models: Past, present and futureAI Open, 2021
- GPU-NEST: Characterizing Energy Efficiency of Multi-GPU Inference ServersIEEE Computer Architecture Letters, 2020
- MLPerf: An Industry Standard Benchmark Suite for Machine Learning PerformanceIEEE Micro, 2020
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015