Pose-aware Large Language Model Interface for Providing Feedback to Sign Language Learners
Open Access
- 27 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Abstract
Sign language learners often find it challenging to self-identify and correct mistakes, and so many turn to automated methods that provide sign language feedback. However, they find that existing methods either require specialized equipment or lack robustness. They, therefore, have to seek human tutors or give up on the inquiry altogether. To overcome the barriers in accessibility and robustness, we build a large language model (LLM)-based tool for that provide feedback to sign language learners. The tool can analyze videos from diverse camera and background settings without specialized equipment thanks to a sign language segmentation and keyframe identification model. Using a pose-aware LLM, the tool can then produce feedback in written language. We present our tool as a demo web application, opening its implementation into specialized learning applications.Keywords
This publication has 12 references indexed in Scilit:
- ChatPose: Chatting about 3D Human Pose2024
- Leveraging Large Language Models With Vocabulary Sharing For Sign Language Translation2023
- SignExplainer: An Explainable AI-Enabled Framework for Sign Language Recognition With Ensemble LearningIEEE Access, 2023
- Sign Language Segmentation with Temporal Convolutional Networks2021
- Sign Segmentation with Changepoint-Modulated Pseudo-Labelling2021
- Student and Teacher Perspectives of Learning ASL in an Online Setting2020
- Teaching deaf culture in American Sign Language courses: Toward a critical pedagogyForeign Language Annals, 2020
- Evaluation of Language Feedback Methods for Student Videos of American Sign LanguageACM Transactions on Accessible Computing, 2017
- American sign language vocabulary2008
- Transforming e‐assessment in American Sign Language: pedagogical and technological enhancements in online language learning and performance assessmentLearning, Media and Technology, 2008