Blind VQA on 360° Video via Progressively Learning From Pixels, Frames, and Video
- 7 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 128-143
Abstract
Blind visual quality assessment (BVQA) on 360° video plays a key role in optimizing immersive multimedia systems. When assessing the quality of 360° video, human tends to perceive its quality degradation from the viewport-based spatial distortion of each spherical frame to motion artifact across adjacent frames, ending with the video-level quality score, i.e., a progressive quality assessment paradigm. However, the existing BVQA approaches for 360° video neglect this paradigm. In this paper, we take into account the progressive paradigm of human perception towards spherical video quality, and thus propose a novel BVQA approach (namely ProVQA) for 360° video via progressively learning from pixels, frames and video. Corresponding to the progressive learning of pixels, frames and video, three sub-nets are designed in our ProVQA approach, i.e., the spherical perception aware quality prediction (SPAQ), motion perception aware quality prediction (MPAQ) and multi-frame temporal non-local (MFTN) sub-nets. The SPAQ sub-net first models the spatial quality degradation based on spherical perception mechanism of human. Then, by exploiting motion cues across adjacent frames, the MPAQ sub-net properly incorporates motion contextual information for quality assessment on 360° video. Finally, the MFTN sub-net aggregates multi-frame quality degradation to yield the final quality score, via exploring long-term quality correlation from multiple frames. The experiments validate that our approach significantly advances the state-of-the-art BVQA performance on 360° video over two datasets, the code of which has been public in https://github.com/yanglixiaoshen/ProVQA .Keywords
Funding Information
- Beijing Natural Science Foundation (JQ20020)
- NSFC (62250001, 62231002, 61922009, 61876013, 61925105)
This publication has 71 references indexed in Scilit:
- On the accuracy of objective image and video quality models: New methodology for performance evaluation2016
- An Optical Flow-Based Full Reference Video Quality Assessment AlgorithmIEEE Transactions on Image Processing, 2016
- Spatiotemporal Statistics for Video Quality AssessmentIEEE Transactions on Image Processing, 2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- A Completely Blind Video Integrity OracleIEEE Transactions on Image Processing, 2015
- A Framework to Evaluate Omnidirectional Video Coding Schemes2015
- No-Reference Video Quality Assessment With 3D Shearlet Transform and Convolutional Neural NetworksIEEE Transactions on Circuits and Systems for Video Technology, 2015
- A Feature-Enriched Completely Blind Image Quality EvaluatorIEEE Transactions on Image Processing, 2015
- Blind Prediction of Natural Video QualityIEEE Transactions on Image Processing, 2014
- No-Reference Image Quality Assessment in the Spatial DomainIEEE Transactions on Image Processing, 2012