Contrastive Self-Supervised Pre-Training for Video Quality Assessment
- 7 December 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 31 (10577149) , 458-471
Abstract
Video quality assessment (VQA) task is an ongoing small sample learning problem due to the costly effort required for manual annotation. Since existing VQA datasets are of limited scale, prior research tries to leverage models pre-trained on ImageNet to mitigate this kind of shortage. Nonetheless, these well-trained models targeting on image classification task can be sub-optimal when applied on VQA data from a significantly different domain. In this paper, we make the first attempt to perform self-supervised pre-training for VQA task built upon contrastive learning method, targeting at exploiting the plentiful unlabeled video data to learn feature representation in a simple-yet-effective way. Specifically, we implement this idea by first generating distorted video samples with diverse distortion characteristics and visual contents based on the proposed distortion augmentation strategy. Afterwards, we conduct contrastive learning to capture quality-aware information by maximizing the agreement on feature representations of future frames and their corresponding predictions in the embedding space. In addition, we further introduce distortion prediction task as an additional learning objective to push the model towards discriminating different distortion categories of the input video. Solving these prediction tasks jointly with the contrastive learning not only provides stronger surrogate supervision signals, but also learns the shared knowledge among the prediction tasks. Extensive experiments demonstrate that our approach sets a new state-of-the-art in self-supervised learning for VQA task. Our results also underscore that the learned pre-trained model can significantly benefit the existing learning based VQA models. Source code is available at https://github.com/cpf0079/CSPT .Keywords
Funding Information
- National Natural Science Foundation of China (62171340, 61771473, 61991451, 61379143)
- Key Project of Shaanxi Provincial Department of Education (20JY024)
- Fundamental Research Funds for the Central Universities (JBF211902)
- Six Talent Peaks High-Level Talents in Jiangsu Province (XYDXX-063)
This publication has 50 references indexed in Scilit:
- Colorful Image Colorization2016
- Context Encoders: Feature Learning by Inpainting2016
- CVD2014—A Database for Evaluating No-Reference Video Quality Assessment AlgorithmsIEEE Transactions on Image Processing, 2016
- Rank Pooling for Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Unsupervised Visual Representation Learning by Context Prediction2015
- A Completely Blind Video Integrity OracleIEEE Transactions on Image Processing, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Blind Prediction of Natural Video QualityIEEE Transactions on Image Processing, 2014
- ViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slicesJournal of Electronic Imaging, 2014
- Study of Subjective and Objective Quality Assessment of VideoIEEE Transactions on Image Processing, 2010