What you get is whatyousee: revisiting the evaluator effect in usability tests
Authors:
- 16 April 2013
- journal article
- research article
- Published by Taylor & Francis in Behaviour & Information Technology
- Vol. 33 (2) , 144-162
Abstract
Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it.Keywords
This publication has 23 references indexed in Scilit:
- A professional empiricist manifestoInteractions, 2011
- A Study of the Evaluator Effect in Usability TestingHuman–Computer Interaction, 2008
- Comparative usability evaluation (CUE-4)Behaviour & Information Technology, 2008
- Metaphors of human thinking for usability inspection and designACM Transactions on Computer-Human Interaction, 2008
- Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on DesignInternational Journal of Human–Computer Interaction, 2006
- The Rashomon EffectSociological Methods & Research, 2002
- Introduction: Current Issues in Usability EvaluationInternational Journal of Human–Computer Interaction, 2001
- Prioritizing usability problems: Data-driven and judgement-driven severity estimatesBehaviour & Information Technology, 2000
- Tracking the effectiveness of usability evaluation methodsBehaviour & Information Technology, 1997
- The Rashomon Effect: When Ethnographers DisagreeAmerican Anthropologist, 1988