Usability Evaluation Ecological Validity: Is More Always Better?
Open Access
- 16 July 2024
- journal article
- research article
- Published by MDPI AG in Healthcare
- Vol. 12 (14) , 1417
Abstract
Background: The ecological validity associated with usability testing of health information technologies (HITs) can affect test results and the predictability of real-world performance. It is, therefore, necessary to identify conditions with the greatest effect on validity. Method: We conducted a comparative analysis of two usability testing conditions. We tested a HIT designed for anesthesiologists to detect pain signals and compared two fidelity levels of ecological validity. We measured the difference in the number and type of use errors identified between high and low-fidelity experimental conditions. Results: We identified the same error types in both test conditions, although the number of errors varied as a function of the condition. The difference in total error counts was relatively modest and not consistent across levels of severity. Conclusions: Increasing ecological validity does not invariably increase the ability to detect use errors. Our findings suggest that low-fidelity tests are an efficient way to identify and mitigate usability issues affecting ease of use, effectiveness, and safety. We believe early low-fidelity testing is an efficient but underused way to maximize the value of usability testing.Keywords
Funding Information
- Agence Nationale de la Recherche (ANR-15-CE36–0007)
This publication has 45 references indexed in Scilit:
- Detectability of use errors in summative usability tests of medical devices: Impact of the test environmentApplied Ergonomics, 2024
- Dimensions of ecological validity for usability evaluations in clinical settingsJournal of Biomedical Informatics, 2020
- Extra-laboratorial usability tests: An empirical comparison of remote and classical field testing with lab testingApplied Ergonomics, 2019
- French legal approach to clinical researchAnaesthesia Critical Care & Pain Medicine, 2018
- The effects of prototype medium on usability testingApplied Ergonomics, 2013
- Methodological issues in product evaluation: The influence of testing environment and task scenarioApplied Ergonomics, 2011
- The influence of user expertise and prototype fidelity in usability testsApplied Ergonomics, 2010
- The influence of prototype fidelity and aesthetics of design in usability tests: Effects on user behaviour, subjective evaluation and emotionApplied Ergonomics, 2009
- Designing interactive consumer products: Utility of paper prototypes and effectiveness of enhanced control labellingApplied Ergonomics, 2007
- Cognitive and usability engineering methods for the evaluation of clinical information systemsJournal of Biomedical Informatics, 2004