AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience
Authors:
- Jing LiJLJing Li
- Lukas KrasulaLKLukas Krasula
- Yoann BaveyeYBYoann Baveye
- Zhi LiZLZhi Li
- Patrick Le CalletPCPatrick Le Callet
- 7 March 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 21 (10) , 2589-2602
Abstract
User expectations have crucial impact on the levels of Quality of Experience (QoE) that they consider acceptable or satisfying. Measuring acceptability and annoyance has mainly been performed in separate or multi-step experiments without any control over participants' expectations. This paper introduces a simple methodology to obtain the information about both of the entities in a single step and compares several data processing strategies useful for results interpretation. A specifically designed subjective experiment, conducted on compressed videos, has shown that the multi-step procedures could be replaced by our proposed single-step approach, regardless of the viewing conditions, while the novel approach is significantly preferred by observers for its low time requirements and higher intuitiveness. The test has simultaneously proven that user expectations can be altered by the instructions and it is therefore possible to simulate different user profiles regardless the participants' real habits. The acceptability/annoyance experimental results are also used to benchmark the state-of-the-art objective video quality metrics in predicting acceptability/annoyance of QoE. A case study on the determination of the threshold of acceptability/annoyance for objective quality metrics is conducted, which can be served as a guideline for video streaming service providers.Keywords
This publication has 32 references indexed in Scilit:
- EXPLORATION AND OPTIMIZATION OF USER EXPERIENCE IN VIEWING VIDEOS ON A MOBILE PHONEInternational Journal of Software Engineering and Knowledge Engineering, 2010
- The big picture on small screens delivering acceptable video quality in mobile TVACM Transactions on Multimedia Computing, Communications, and Applications, 2009
- How low can you go? The effect of low resolutions on shot types in mobile TVMultimedia Tools and Applications, 2007
- A new standardized method for objectively measuring video qualityIEEE Transactions on Broadcasting, 2004
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Note on the Sampling Error of the Difference Between Correlated Proportions or PercentagesPsychometrika, 1947
- The Generalization of `Student's' Problem when Several Different Population Variances are InvolvedBiometrika, 1947
- A New Test for 2 × 2 TablesNature, 1945
- A law of comparative judgment.Psychological Review, 1927
- On the Interpretation of χ 2 from Contingency Tables, and the Calculation of PJournal of the Royal Statistical Society, 1922