Duk Shin

The development of automatic fingerspelling recognition systems is critical for facilitating seamless communication for the deaf and hard-of-hearing community. Historically, recognition technologies relied on wearable sensors, such as data gloves, inertial measurement units (IMUs), and surface electromyography (sEMG), which provided stable kinematic data but suffered from high hardware costs and user discomfort. Non-invasive vision-based sensing has emerged to eliminate wearable overhead but is subject to severe physical bottlenecks: continuous high-speed movements inevitably introduce motion blur and self-occlusion, drastically degrading tracking accuracy. To address these visual sensing constraints, we investigate a visual-language integration framework that combines dynamic visual representations derived from a single RGB video stream with the lexical knowledge of Large Language Models (LLMs), allowing the language model to compensate for uncertainty or errors in the visual representation. Our approach employs a robust visual encoder—comprising a Temporal Convolutional Network (TCN) and a Transformer—to extract and integrate dynamic transition features from 3D skeletal coordinates. Rather than propose a new LLM architecture, this study focuses on how visual information should be represented and transferred to an LLM by comparing discrete-text, probability-level, and latent-feature-level integration strategies under the same experimental conditions. Specifically, we systematically evaluated three integration strategies: Text Input (TI), Projector Probabilities Input (PPI), and Projector Transformer Features Input (PTFI). Experimental results on a 28-word closed-set evaluation show that while the word-level exact-match accuracy (WA) of the visual encoder alone was limited to 11.66% with continuous signing, our LLM-integrated architecture elevated the WA to a maximum of 87.73%. Crucially, the PTFI approach successfully suppressed character-level degradation by grounding linguistic reasoning in latent Transformer-based visual evidence. These results suggest that integrating LLM-based linguistic reasoning with dynamic visual features is a promising approach for closed-set continuous fingerspelling recognition under controlled experimental conditions.

Japanese fingerspelling recognition faces unique challenges due to rapid temporal transitions and subtle hand shape variations difficult to capture with conventional methods. We address these challenges by applying Hierarchical Reliability Propagation (HR-Pro), a point-supervised Temporal Action Localization (TAL) method that requires only sparse point-level annotations instead of dense frame-wise labels. We enhance the HR-Pro framework through three key innovations: (1) replacing the I3D encoder with pretrained VideoMAE v2 for superior temporal dynamics modeling without optical flow, (2) introducing point-supervised contrastive learning inspired by SimCLRfor robust feature discrimination, and (3) incorporating 20-dimensional joint angle features from MediaPipe for explicit kinematic modeling While the dataset used in this work has been employed internally, we formally introduce it as a public resource featuring continuous and isolated fingerspelling sequences with diverse phonetic coverage and signer variation. Our top model achieves up to 93.4% mAP at tIoU 0.1-0.5, using a pretrained VideoMAE v2 enhanced with our point-supervised contrastive learning, outperforming I3D-based baselines while reducing computational complexity. Notably, incorporating joint angle features with I3D yields over a 30 percentage point improvement, demonstrating the value of kinematic cues. Ablation studies reveal interesting redundancy between angle features and contrastive pretraining, highlighting the importance of modality-aware fusion strategies. Code is available at: https://github.com/tpu-kanglabs/ub-hrpro

To improve the accuracy of Japanese finger-spelled character recognition using an RGB camera, we focused on feature design and refinement of the recognition method. By leveraging angular features extracted via MediaPipe, we proposed a method that effectively captures subtle motion differences while minimizing the influence of background and surrounding individuals. We constructed a large-scale dataset that includes not only the basic 50 Japanese syllables but also those with diacritical marks, such as voiced sounds (e.g., “ga”, “za”, “da”) and semi-voiced sounds (e.g., “pa”, “pi”, “pu”), to enhance the model’s ability to recognize a wide variety of characters. In addition, the application of a change-point detection algorithm enabled accurate segmentation of sign language motion boundaries, improving word-level recognition performance. These efforts laid the foundation for a highly practical recognition system. However, several challenges remain, including the limited size and diversity of the dataset and the need for further improvements in segmentation accuracy. Future work will focus on enhancing the model’s generalizability by collecting more diverse data from a broader range of participants and incorporating segmentation methods that consider contextual information. Ultimately, the outcomes of this research should contribute to the development of educational support tools and sign language interpretation systems aimed at real-world applications.

Wrist movements play a crucial role in upper-limb motor tasks. As prosthetic and robotic hand technologies have evolved, increasing attention has been focused on replicating the anatomy and functionality of the wrist. Closely imitating the biomechanics and movement mechanisms of human limbs is expected to enhance the overall performance of bionic robotic hands. This study presents the design of a tendon-driven bionic spherical robot wrist, utilizing two pairs of cables that mimic antagonist muscle pairs. The cables are actuated by pulleys driven by servo motors, allowing for two primary wrist motions: flexion–extension and ulnar–radial deviation. The performance Please confirm if the “1583 Iiyama” is necessary. Same as belowof the proposed robot wrist is validated through manipulation experiments using a prototype, demonstrating its capability to achieve a full range of motion for both ulnar and radial deviation. This wrist mechanism is expected to be integrated into robotic systems, enabling greater flexibility and more human-like movement capabilities.

In recent years, deep learning technology has greatly advanced research designed to improve communication with hearing-impaired individuals. This research has particularly focused on improving systems for automatically recognizing and translating sign language. Our study introduces a new method that utilizes a Vision Transformer (ViT) to recognize Japanese Sign Language in real-time. This technique employs the Mediapipe pose estimation library to pinpoint the exact positions of finger joints within video frames and converts these positions into one-dimensional angular features. These features are then organized sequentially to create a two-dimensional input vector for the ViT model. We explored different configurations by varying the number of encoder layers in the ViT model to find the most accurate setup. The proposed method was further tested on video data representing words to assess its effectiveness, covering signs with and without actions. Our experiments with Japanese sign language videos showed the proposed approach's real-time capabilities and accuracy evalutation.

In recent years, developments in deep learning technology have driven significant advancements in research aimed at facilitating communication with individuals who have hearing impairments. The focus has been on enhancing automatic recognition and translation systems for sign language. This study proposes a novel approach using a vision transformer (ViT) for recognizing Japanese Sign Language. Our method employs a pose estimation library, MediaPipe, to extract the positional coordinates of each finger joint within video frames and generate one-dimensional angular feature data from these coordinates. Then, the code arranges these feature data in a temporal sequence to form a two-dimensional input vector for the ViT model. To determine the optimal configuration, this study evaluated recognition accuracy by manipulating the number of encoder layers within the ViT model and compared against traditional convolutional neural network (CNN) models to evaluate its effectiveness. The experimental results showed 99.7% accuracy for the method using the ViT model and 99.3% for the results using the CNN. We demonstrated the efficacy of our approach through real-time recognition experiments using Japanese sign language videos.

Tensegrity is a structure consisting of rigid bodies and internal tensile members, with no contact between the rigid bodies. The model of an arm with a tensegrity structure is not movable as it is, but we believe that it can be made movable and flexible by incorporating springs. We developed an arm that incorporates springs in the arm’s tensile members by extending the model of an arm with a tensegrity structure. Then, as an evaluation of the developed arm, we measured the ranges of motions and the forces required for that motion. We also developed a mechanism that allows the arm to bend and extend. We believe that this method of making the tensegrity arm controllable by incorporating springs will be useful in the development of flexible robotic arms for caregiving using robots and other applications.

Many patients with upper limb defects desire myoelectric prosthetic hands, but they are still not used for some reasons. One of the most significant reasons is its external appearance, which has the discomfort caused by the structural difference between a human hand and a robotic link. The structure must be based on human anatomy to create a more natural-looking prosthesis. This study designed a biomimetic prosthetic hand with bones, ligaments, tendons, and multiple muscles based on the human musculoskeletal system. We verified the proposed prosthetic hand using the viscoelastic angle sensor to determine whether it works like a human hand. We also compared the finger force of the prosthetic hand with that of a human finger. It could be capable of controlling the angle and the stiffness of the joint by multiple extensor and flexor muscles, like humans.