Validation of system usability scale as a usability metric to evaluate voice user interfaces
Open Access
- 29 February 2024
- journal article
- research article
- Published by PeerJ in PeerJ Computer Science
- Vol. 10, e1918
Abstract
In recent years, user experience (UX) has gained importance in the field of interactive systems. To ensure its success, interactive systems must be evaluated. As most of the standardized evaluation tools are dedicated to graphical user interfaces (GUIs), the evaluation of voice-based interactive systems or voice user interfaces is still in its infancy. With the help of a well-established evaluation scale, the System Usability Scale (SUS), two prominent, widely accepted voice assistants were evaluated. The evaluation, with SUS, was conducted with 16 participants who performed a set of tasks on Amazon Alexa Echo Dot and Google Nest Mini. We compared the SUS score of Amazon Alexa Echo Dot and Google Nest Mini. Furthermore, we derived the confidence interval for both voice assistants. To enhance understanding for usability practitioners, we analyzed the Adjective Rating Score of both interfaces to comprehend the experience of an interface’s usability through words rather than numbers. Additionally, we validated the correlation between the SUS score and the Adjective Rating Score. Finally, a paired sample t-test was conducted to compare the SUS score of Amazon Alexa Echo Dot and Google Nest Mini. This resulted in a huge difference in scores. Hence, in this study, we corroborate the utility of the SUS in voice user interfaces and conclude by encouraging researchers to use SUS as a usability metric to evaluate voice user interfaces.Keywords
This publication has 23 references indexed in Scilit:
- Development and user evaluation for a robust character input support system for natural images of electricity reading slipsElectronics and Communications in Japan, 2023
- Psychometric Evaluation of the F-SUS: Creation and Validation of the French Version of the System Usability ScaleInternational Journal of Human–Computer Interaction, 2021
- The Adoption Analysis of Voice-Based Smart IoT ProductsIEEE Internet of Things Journal, 2020
- Pilot study of a newly developed eLearning tool to teach CT and PET/CT in radiology and nuclear medicineNuklearmedizin-Nuclear Medicine, 2020
- Hey Alexa … examine the variables influencing the use of artificial intelligent in-home voice assistantsComputers in Human Behavior, 2019
- The System Usability Scale: Past, Present, and FutureInternational Journal of Human–Computer Interaction, 2018
- An Empirical Evaluation of the System Usability ScaleInternational Journal of Human–Computer Interaction, 2008
- The Basic Practice of StatisticsTechnometrics, 1996
- IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for useInternational Journal of Human–Computer Interaction, 1995
- PERCEIVED USEFULNESS OF INFORMATION: A PSYCHOMETRIC EXAMINATION*Decision Sciences, 1980