Group MAD Competition? A New Methodology to Compare Objective Image Quality Models
Authors:
- Kede MaKMKede Ma
- Qingbo WuQWQingbo Wu
- Zhou WangZWZhou Wang
- Zhengfang DuanmuZDZhengfang Duanmu
- Hongwei YongHYHongwei Yong
- Hongliang LiHLHongliang Li
- Lei ZhangLZLei Zhang
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1664-1673
Abstract
Objective image quality assessment (IQA) models aim to automatically predict human visual perception of image quality and are of fundamental importance in the field of image processing and computer vision. With an increasing number of IQA models proposed, how to fairly compare their performance becomes a major challenge due to the enormous size of image space and the limited resource for subjective testing. The standard approach in literature is to compute several correlation metrics between subjective mean opinion scores (MOSs) and objective model predictions on several well-known subject-rated databases that contain distorted images generated from a few dozens of source images, which however provide an extremely limited representation of real-world images. Moreover, most IQA models developed on these databases often involve machine learning and/or manual parameter tuning steps to boost their performance, and thus their generalization capabilities are questionable. Here we propose a novel methodology to compare IQA models. We first build a database that contains 4,744 source natural images, together with 94,880 distorted images created from them. We then propose a new mechanism, namely group MAximum Differentiation (gMAD) competition, which automatically selects subsets of image pairs from the database that provide the strongest test to let the IQA models compete with each other. Subjective testing on the selected subsets reveals the relative performance of the IQA models and provides useful insights on potential ways to improve them. We report the gMAD competition results between 16 well-known IQA models, but the framework is extendable, allowing future IQA models to be added into the competition.Keywords
This publication has 34 references indexed in Scilit:
- Image database TID2013: Peculiarities, results and perspectivesSignal Processing: Image Communication, 2015
- Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian FeaturesIEEE Transactions on Image Processing, 2014
- Is Crowdsourcing for Optical Flow Ground Truth Generation Feasible?Lecture Notes in Computer Science, 2013
- VCL@FER Image Quality Assessment DatabaseAutomatika, 2012
- Statistical ranking and combinatorial Hodge theoryMathematical Programming, 2010
- Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantitiesJournal of Vision, 2008
- Modern Image Quality AssessmentSynthesis Lectures on Image, Video, and Multimedia Processing, 2006
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Adaptive thermal comfort and sustainable thermal standards for buildingsEnergy and Buildings, 2002
- Picture Processing by ComputerACM Computing Surveys, 1969