Nora Dunphy

Intelligent Personal Assistants (IPAs) are currently limited to mostly voice input by users, which often does not work for Deaf and Hard of Hearing (DHH) users’ accessibility. Touch interfaces are an accessible alternative in principle, and these recently have been combined with large language models (LLMs) for usability enhancements. However, these are not hands-free, and it is not always feasible to walk up to a device and interact with its touchscreen, such as in the kitchen with dirty hands. This paper situates an LLM-powered touch interface against hands-free options. We present a study with 23 DHH participants who tested three potential input methods for interacting with IPAs: American Sign Language (ASL) in a Wizard-of-Oz setting, LLM-assisted touch through a touchscreen, and LLM-assisted touch through hands-free head pointing. ASL and LLM-assisted touch had comparable usability scores, while headpointing scored much worse. Despite comparable usability between ASL and touch, participants were much more enthusiastic about ASL input. This suggests ASL recognition should be the ultimate goal, but because such technology is not yet commercially viable, further research is needed for identifying practical hands-free alternatives to voice interaction with IPAs.