Effect of Automatic Sign Recognition Performance on the Usability of Video-Based Search Interfaces for Sign Language Dictionaries
Open Access
- 24 October 2019
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 56-67
Abstract
Researchers have investigated various methods to help users search for the meaning of an unfamiliar word in American Sign Language (ASL). Some are based on sign-recognition technology, e.g. a user performs a word into a webcam and obtains a list of possible matches in the dictionary. However, developers of such technology report the performance of their systems inconsistently, and prior research has not examined the relationship between the performance of search technology and users' subjective judgements for this task. We conducted two studies using a Wizard-of-Oz prototype of a webcam-based ASL dictionary search system to investigate the relationship between the performance of such a system and user judgements. We found that in addition to the position of the desired word in a list of results, which is what is often reported in literature; the similarity of the other words in the results list also affected users' judgements of the system. We also found that metrics that incorporate the precision of the overall list correlated better with users' judgements than did metrics currently reported in prior ASL dictionary research.Keywords
Funding Information
- National Science Foundation (1763569)
This publication has 16 references indexed in Scilit:
- ASL-LEX: A lexical database of American Sign LanguageBehavior Research Methods, 2016
- Correlation Between System and User Metrics in a Session2016
- An integrated RGB-D system for looking up the meaning of signs2015
- A User-Powered American Sign Language Dictionary2015
- Reading the signs: A video based sign dictionary2011
- Rapid signer adaptation for continuous sign language recognition using a combined approach of eigenvoices, MLLR, and MAP2008 19th International Conference on Pattern Recognition, 2008
- How well does result relevance predict session satisfaction?2007
- The relationship between IR effectiveness measures and user satisfaction2007
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- A taxonomy of web searchACM SIGIR Forum, 2002