Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL Learners
Authors:
- Saad HassanSHSaad Hassan
- Caluã de Lacerda PatacaCPCaluã de Lacerda Pataca
- Akhter Al AminAAAkhter Al Amin
- Laleh NourianLNLaleh Nourian
- Diego NavarroDNDiego Navarro
- Sooyeon LeeSLSooyeon Lee
- Alexis GordonAGAlexis Gordon
- Matthew WatkinsMWMatthew Watkins
- Garreth W. TigwellGTGarreth W. Tigwell
- Matt HuenerfauthMHMatt Huenerfauth
Open Access
- 30 September 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Accessible Computing
- Vol. 17 (3) , 1-35
Abstract
People learning American Sign Language (ASL) and practicing their comprehension skills will often encounter complex ASL videos that may contain unfamiliar signs. Existing dictionary tools require users to isolate a single unknown sign before initiating a search by selecting linguistic properties or performing the sign in front of a webcam. This process presents challenges in extracting and reproducing unfamiliar signs, disrupting the video-watching experience, and requiring learners to rely on external dictionaries. We explore a technology that allows users to select and view dictionary results for one or more unfamiliar signs while watching a video. We interviewed 14 ASL learners to understand their challenges in understanding ASL videos, strategies for dealing with unfamiliar vocabulary, and expectations for an in situ dictionary system. We then conducted an in-depth analysis with eight learners to examine their interactions with a Wizard-of-Oz prototype during a video comprehension task. Finally, we conducted a comparative study with six additional ASL learners to evaluate the speed, accuracy, and workload benefits of an embedded dictionary-search feature within a video player. Our tool outperformed a baseline in the form of an existing online dictionary across all three metrics. The integration of a search tool and span selection offered advantages for video comprehension. Our findings have implications for designers, computer vision researchers, and sign language educators.Keywords
Funding Information
- National Science Foundation (1763569, 2212303, and 2125362)
This publication has 52 references indexed in Scilit:
- Sign Language Recognition: A Deep SurveyExpert Systems with Applications, 2021
- Kahaniyan - Designing for Acquisition of Urdu as a Second Language2019
- A Modified LSTM Model for Continuous Sign Language Recognition Using Leap MotionIEEE Sensors Journal, 2019
- Approaches to Human and Machine Translation Quality Assessment2018
- A position and rotation invariant framework for sign language recognition (SLR) using KinectMultimedia Tools and Applications, 2017
- Whose ASL counts? Linguistic prescriptivism and challenges in the context of parent sign language curriculum developmentInternational Journal of Bilingual Education and Bilingualism, 2016
- Translating Video: Obstacles and ChallengesProcedia - Social and Behavioral Sciences, 2014
- A study of sign language coarticulationACM SIGACCESS Accessibility and Computing, 2009
- Eye Gaze During Comprehension of American Sign Language by Native and Beginning SignersJournal of Deaf Studies and Deaf Education, 2008
- American Sign Language compound formation processes, lexicalization, and phonological remnantsNatural Language & Linguistic Theory, 1986