Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
Preprint
Abstract
Due to increasing amounts of data and compute resources, deep learning achieves many successes in various domains. The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem to be solved. Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices. AIoTBench covers three typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is implemented by three frameworks which are designed for mobile and embedded devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI capabilities of the devices, we propose two unified metrics as the AI scores: Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we have compared and ranked 5 mobile devices using our benchmark. This list will be extended and updated soon after.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- MLPerf Inference Benchmark2020
- AIoT Bench: Towards Comprehensive Benchmarking Mobile and Embedded Device Intelligence2019
- AI Benchmark: All About Deep Learning on Smartphones in 20192019
- Searching for MobileNetV32019
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- Benchmark Analysis of Representative Deep Neural Network ArchitecturesIEEE Access, 2018
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Identity Mappings in Deep Residual Networks2016
- Microsoft COCO: Common Objects in Context2014