Clarifying Agreement Calculations and Analysis for End-User Elicitation Studies
Open Access
- 7 January 2022
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Computer-Human Interaction
- Vol. 29 (1) , 1-70
Abstract
We clarify fundamental aspects of end-user elicitation, enabling such studies to be run and analyzed with confidence, correctness, and scientific rigor. To this end, our contributions are multifold. We introduce a formal model of end-user elicitation in HCI and identify three types of agreement analysis:expert,codebook, andcomputer. We show that agreement is a mathematicaltolerance relationgenerating a tolerance space over the set of elicited proposals. We review current measures of agreement and show that all can be computed from anagreement graph. In response to recent criticisms, we show that chance agreement represents an issue solely for inter-rater reliability studies and not for end-user elicitation, where it is opposed bychance disagreement. We conduct extensive simulations of 16 statistical tests for agreement rates, and report Type I errors and power. Based on our findings, we provide recommendations for practitioners and introduce a five-level hierarchy for elicitation studies.Keywords
This publication has 115 references indexed in Scilit:
- Iconic and multi-stroke gesture recognitionPattern Recognition, 2009
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- A guided tour to approximate string matchingACM Computing Surveys, 2001
- Participatory designCommunications of the ACM, 1993
- Specifying gestures by exampleACM SIGGRAPH Computer Graphics, 1991
- Diversity as a Concept and its MeasurementJournal of the American Statistical Association, 1982
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- Reliability of Content Analysis: The Case of Nominal Scale CodingPublic Opinion Quarterly, 1955