Development of an Animation-Based Indian Sign Language e-Learning App for Persons With Hearing and Speech Impairment
Open Access
- 24 November 2024
- journal article
- Published by Springer Nature in Cureus Journal of Computer Science
- Vol. 16 (11)
Abstract
Human-computer interaction (HCI) is changing the quality of our lives. In this paper, we present one such application of HCI that comes as a boon in the life of a hearing-impaired person. We propose to build an e-learning tool for the education of the hearing- and speech-impaired community by creating animated gestures in sign language, with particular emphasis on the Indian Sign Language (ISL). With the help of MATLAB App Designer, we developed a standalone MATLAB application for ISL fingerspelling (including digits, alphabets and combination of these two) and some ISL words of daily usage. This app can be used by the hearing-impaired community for learning ISL, even while seated at home, or for training signers and teachers at special schools. In our proposed method, the software first learns different sign gestures from real gesture videos. For this, hand regions in every frame are extracted using image processing algorithms and subsequently identifying key video frames that show significant change in hand shape and/or position during gesturing. Correlation coefficient and structural similarity are the two metrics used for the purpose. Next, several hand parameters such as the finger joint angles and orientation of the palm in these key frames are derived using Google MediaPipe holistic model. When using the app, for a query sign, the corresponding gesture is animated by synthesizing the key frames containing 3D hand shapes with desired palm orientation and finger joint angles using the stored hand parameters, followed by interpolating the in-between frames using image metamorphosis. For interpolation, we have used spline interpolation and intensity interpolation. Some sign gestures created in our experiments show that our proposed method generates smooth and natural-looking animated videos from a comparatively less amount of stored gesture information, thereby offering large savings in memory.Keywords
This publication has 28 references indexed in Scilit:
- Motion Transformer for Unsupervised Image Animation2022
- X2Face: A Network for Controlling Face Generation Using Images, Audio, and Pose Codes2018
- Hand gesture recognition and animation for local hand motionsInternational Journal of Machine Learning and Cybernetics, 2013
- Design, development and field evaluation of a Spanish into sign language translation systemPattern Analysis and Applications, 2011
- Embodied conversational agents in computer assisted language learningSpeech Communication, 2009
- Implementing Expressive Gesture Synthesis for Embodied Conversational Agents2006
- Synthesizing multimodal utterances for conversational agentsComputer Animation and Virtual Worlds, 2004
- Gesture Synthesis in a Real-World ECALecture Notes in Computer Science, 2004
- Synthetic Animation of Deaf Signing GesturesLecture Notes in Computer Science, 2002
- Image morphing: a surveyThe Visual Computer, 1998