Electromyogram‐Based Lip‐Reading via Unobtrusive Dry Electrodes and Machine Learning Methods
Abstract
Lip‐reading provides an effective speech communication interface for people with voice disorders and for intuitive human–machine interactions. Existing systems are generally challenged by bulkiness, obtrusiveness, and poor robustness against environmental interferences. The lack of a truly natural and unobtrusive system for converting lip movements to speech precludes the continuous use and wide‐scale deployment of such devices. Here, the design of a hardware–software architecture to capture, analyze, and interpret lip movements associated with either normal or silent speech is presented. The system can recognize different and similar visemes. It is robust in a noisy or dark environment. Self‐adhesive, skin‐conformable, and semi‐transparent dry electrodes are developed to track high‐fidelity speech‐relevant electromyogram signals without impeding daily activities. The resulting skin‐like sensors can form seamless contact with the curvilinear and dynamic surfaces of the skin, which is crucial for a high signal‐to‐noise ratio and minimal interference. Machine learning algorithms are employed to decode electromyogram signals and convert them to spoken words. Finally, the applications of the developed lip‐reading system in augmented reality and medical service are demonstrated, which illustrate the great potential in immersive interaction and healthcare applications.Keywords
Funding Information
- National Science Foundation (2129673)
This publication has 48 references indexed in Scilit:
- A review of recent advances in visual speech decodingImage and Vision Computing, 2014
- Materials and Optimized Designs for Human‐Machine Interfaces Via Epidermal ElectronicsAdvanced Materials, 2013
- THE FURHAT BACK-PROJECTED HUMANOID HEAD–LIP READING, GAZE AND MULTI-PARTY INTERACTIONInternational Journal of Humanoid Robotics, 2013
- Small-vocabulary speech recognition using a silent speech interface based on magnetic sensingSpeech Communication, 2013
- Lip-reading and the ventilated patient*Critical Care Medicine, 2012
- Mechanics of Epidermal ElectronicsJournal of Applied Mechanics, 2012
- Modeling coarticulation in EMG-based continuous speech recognitionSpeech Communication, 2010
- Hearing Rehabilitation in Neurofibromatosis Type 2 Patients: Cochlear versus Auditory Brainstem ImplantationAudiology and Neurotology, 2008
- Chronic neural recordings using silicon microelectrode arrays electrochemically deposited with a poly(3,4-ethylenedioxythiophene) (PEDOT) filmJournal of Neural Engineering, 2006
- Audio-visual speech modeling for continuous speech recognitionIEEE Transactions on Multimedia, 2000