KUCHIMOJI: A Japanese Vowel-Based Character Entry System Using Mouth Shape Recognition for Assistive Communication

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Patients with neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS) frequently lose the ability to communicate through speech or writing. However, their cognitive and sensory functions are often relatively preserved. In Japan, the traditional method known as kuchimoji (mouth-based character communication) enables character-by-character communication using mouth shapes. This method relies heavily on caregiver skill and is challenging to implement consistently. This study introduces KUCHIMOJI, a Japanese text input system that uses mouth-shape recognition to support independent augmentative and alternative communication (AAC) without caregiver assistance. The system employs a lightweight convolutional neural network (MobileNetV2) to classify six mouth shapes. These shapes correspond to five vowels and a closed-lip state. To accommodate diverse user conditions, a multimodal input framework is designed. It supports three operation modes: facial-image-based signal input, button-based input, and key-based direct input. As an initial feasibility study, experiments with ten healthy participants were conducted to evaluate text entry performance in terms of text entry speed (TES) and miss entry rate (MER). Results indicate that the system achieves average input speeds of 3.86, 5.32, and 11.35 characters per minute (cpm) for the facial-image, button, and key-based modes, respectively. It maintains low error rates (2.96–5.05%). These findings suggest that the system offers a flexible trade-off between speed and accuracy depending on the input modality. The proposed approach provides a practical, low-cost, non-contact communication solution. This underscores its potential forpractical assistive communication applications.
Funding Information
  • Japan Society for the Promotion of Science (23H03787)