OmniLive: Super-Resolution Enhanced 360° Video Live Streaming for Mobile Devices
Open Access
- 18 June 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 261-274
Abstract
The live streaming of omnidirectional video (ODV) on mobile devices demands considerable network resources; thus, current mobile networks are incapable of providing users with high-quality ODV equivalent to conventional flat videos. We observe that mobile devices, in fact, underutilize graphics processing units (GPUs) while processing ODVs; hence, we envisage an opportunity exists in exploiting video super-resolution (VSR) for improved ODV quality. However, the device-specific discrepancy in GPU capability and dynamic behavior of GPU frequency in mobile devices create a challenge in providing VSR-enhanced ODV streaming. In this paper, we propose OmniLive, an on-device VSR system for mobile ODV live streaming. OmniLive addresses the dynamicity of GPU capability with an anytime inference-based VSR technique called Omni SR. For Omni SR, we design a VSR deep neural network (DNN) model with multiple exits and an inference scheduler that decides on the exit of the model at runtime. OmniLive also solves the performance heterogeneity of mobile GPUs using the Omni neural architecture search (NAS) scheme. Omni NAS finds an appropriate DNN model for each mobile device with Omni SR-specific neural architecture search techniques. We implemented OmniLive as a fully functioning system encompassing a streaming server and Android application. The experiment results show that our anytime VSR model provides four times upscaled videos while saving up to 57.15% of inference time compared with the previous super-resolution model showing the lowest inference time on mobile devices. Moreover, OmniLive can maintain 30 frames per second while fully utilizing GPUs on various mobile devices.Keywords
Funding Information
- Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (2018-0-00532)
This publication has 30 references indexed in Scilit:
- Regularized Evolution for Image Classifier Architecture SearchProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Video Enhancement with Task-Oriented FlowInternational Journal of Computer Vision, 2019
- Flare2018
- Detail-Revealing Deep Video Super-Resolution2017
- Densely Connected Convolutional Networks2017
- Fixation Prediction for 360° Video Streaming in Head-Mounted Virtual Reality2017
- Deep Residual Learning for Image Recognition2016
- Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network2016
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- On Bayesian Adaptive Video Super ResolutionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013