Usability Problem Reports for Comparative Studies: Consistency and Inspectability
Open Access
- 5 December 2008
- journal article
- research article
- Published by Taylor & Francis in Human–Computer Interaction
- Vol. 23 (4) , 329-380
Abstract
This study explores issues of consistency and inspectability in usability test data analysis processes and reports. Problem reports resulting from usability tests performed by three professional usability labs in three different countries are compared. Each of the labs conducted a usability test on the same product, applying an agreed test protocol that was collaboratively developed by the labs. Each lab first analyzed their own findings as they always do in their regular professional practice. A few weeks later, they again analyzed their findings, but then everyone applied the same method (SlimDEVAN: a simplified version of DEVAN, a method developed for facilitating comparison of findings from usability tests in an academic setting). It was found that levels of agreement between labs did not improve when they all used SlimDEVAN, suggesting that there was inherent subjectivity in their analyses. It was found that consistency of single analyst teams varied considerably and that a method like SlimDEVAN can help in making the analysis process and findings more inspectable. Inspectability is helpful in comparative studies based on identified usability problems because it allows for tracing back findings to original observations as well as for laying bare the subjective parts of the data analysis.Keywords
This publication has 9 references indexed in Scilit:
- Developing a coding scheme for detecting usability and fun problems in computer games for young childrenBehavior Research Methods, 2006
- Cultural differences on attention and perceived usability: Investigating color combinations of animated graphicsInternational Journal of Human-Computer Studies, 2006
- Comparative usability evaluationBehaviour & Information Technology, 2004
- Cognitive, physical, sensory, and functional affordances in interaction designBehaviour & Information Technology, 2003
- DEVAN: A tool for detailed video analysis of user test dataBehaviour & Information Technology, 2002
- The Evaluator Effect: A Chilling Fact About Usability Evaluation MethodsInternational Journal of Human–Computer Interaction, 2001
- Criteria For Evaluating Usability Evaluation MethodsInternational Journal of Human–Computer Interaction, 2001
- A structured methodology for comparative evaluation of user interface designs using usability criteria and measuresInternational Journal of Industrial Ergonomics, 1999
- Comparison of evaluation methods using structured usability problem reportsBehaviour & Information Technology, 1997