Crowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing
Authors:
- 1 June 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15503607,p. 1070-1075
Abstract
Since its introduction a few years ago, the concept of `Crowdsourcing' has been heralded as highly attractive alternative approach towards evaluating the Quality of Experience (QoE) of networked multimedia services. The main reason is that, in comparison to traditional laboratory-based subjective quality testing, crowd-based QoE assessment over the Internet promises to be not only much more cost-effective (no lab facilities required, less cost per subject) but also much faster in terms of shorter campaign setup and turnaround times. However, the reliability of remote test subjects and consequently, the trustworthiness of study results is still an issue that prevents the widespread adoption of crowd-based QoE testing. Various ideas for improving user rating reliability and test efficiency have been proposed, with the majority of them relying on a posteriori analysis of results. However, such methods introduce a major lag that significantly affects efficiency of campaign execution. In this paper we address these shortcomings by introducing in momento methods for crowdsourced video QoE assessment which yield improvements of results reliability by factor two and campaign execution efficiency by factor ten. The proposed in momento methods are applicable to existing crowd-based QoE testing approaches and suitable for a variety of service scenarios.Keywords
This publication has 12 references indexed in Scilit:
- Challenges in crowd-based video quality assessment2012
- Microworkers vs. facebook: The impact of crowdsourcing platform choice on experimental results2012
- Increasing cheat robustness of crowdsourcing tasksInformation Retrieval Journal, 2012
- Quantification of YouTube QoE via Crowdsourcing2011
- Conducting behavioral research on Amazon’s Mechanical TurkBehavior Research Methods, 2011
- Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com2011
- CROWDMOS: An approach for crowdsourcing mean opinion score studies2011
- Are your participants gaming the system?2010
- Crowdsourcing for relevance evaluationACM SIGIR Forum, 2008
- Crowdsourcing user studies with Mechanical Turk2008