Customizing Generated Signs and Voices of AI Avatars: Deaf-Centric Mixed-Reality Design for Deaf-Hearing Communication
Authors:
- Si ChenSCSi Chen
- Haocong ChengHCHaocong Cheng
- Suzy SuSSSuzy Su
- Stephanie PattersonSPStephanie Patterson
- Raja KushalnagarRKRaja Kushalnagar
- Qi WangQWQi Wang
- Yun HuangYHYun Huang
Preprint
Abstract
This study investigates innovative interaction designs for communication and collaborative learning between learners of mixed hearing and signing abilities, leveraging advancements in mixed reality technologies like Apple Vision Pro and generative AI for animated avatars. Adopting a participatory design approach, we engaged 15 d/Deaf and hard of hearing (DHH) students to brainstorm ideas for an AI avatar with interpreting ability (sign language to English, voice to English) that would facilitate their face-to-face communication with hearing peers. Participants envisioned the AI avatars to address some issues with human interpreters, such as lack of availability, and provide affordable options to expensive personalized interpreting service. Our findings indicate a range of preferences for integrating the AI avatars with actual human figures of both DHH and hearing communication partners. The participants highlighted the importance of having control over customizing the AI avatar, such as AI-generated signs, voices, facial expressions, and their synchronization for enhanced emotional display in communication. Based on our findings, we propose a suite of design recommendations that balance respecting sign language norms with adherence to hearing social norms. Our study offers insights on improving the authenticity of generative AI in scenarios involving specific, and sometimes unfamiliar, social norms.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Machine translation from signed to spoken languages: state of the art and challengesUniversal Access in the Information Society, 2023
- Inclusive AR/VR: accessibility barriers for immersive technologiesUniversal Access in the Information Society, 2023
- Holographic sign language avatar interpreter: A user interaction study in a mixed reality classroomComputer Animation and Virtual Worlds, 2022
- Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production2022
- Ask and ye shall not receive: Interpreter-related access barriers reported by Deaf users of American sign languageDisability and Health Journal, 2020
- Healthcare providers’ and deaf patients’ interpreting preferences for critical care and non-critical care: Video remote interpretingDisability and Health Journal, 2020
- Virtual reality, augmented reality, mixed reality, and the marine conservation movementAquatic Conservation: Marine and Freshwater Ecosystems, 2017
- The impact of input quality on early sign development in native and non-native language learnersJournal of Child Language, 2016
- HamNoSys to SiGML Conversion System for Sign Language AutomationProcedia Computer Science, 2016
- When Parents Are Deaf Versus Hard of Hearing: Patterns of Sign Use and School Placement of Deaf and Hard-of-Hearing ChildrenJournal of Deaf Studies and Deaf Education, 2004