Deep Learning Reader for Visually Impaired
Open Access
- 16 October 2022
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 11 (20) , 3335
Abstract
Recent advances in machine and deep learning algorithms and enhanced computational capabilities have revolutionized healthcare and medicine. Nowadays, research on assistive technology has benefited from such advances in creating visual substitution for visual impairment. Several obstacles exist for people with visual impairment in reading printed text which is normally substituted with a pattern-based display known as Braille. Over the past decade, more wearable and embedded assistive devices and solutions were created for people with visual impairment to facilitate the reading of texts. However, assistive tools for comprehending the embedded meaning in images or objects are still limited. In this paper, we present a Deep Learning approach for people with visual impairment that addresses the aforementioned issue with a voice-based form to represent and illustrate images embedded in printed texts. The proposed system is divided into three phases: collecting input images, extracting features for training the deep learning model, and evaluating performance. The proposed approach leverages deep learning algorithms; namely, Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), for extracting salient features, captioning images, and converting written text to speech. The Convolution Neural Network (CNN) is implemented for detecting features from the printed image and its associated caption. The Long Short-Term Memory (LSTM) network is used as a captioning tool to describe the detected text from images. The identified captions and detected text is converted into voice message to the user via Text-To-Speech API. The proposed CNN-LSTM model is investigated using various network architectures, namely, GoogleNet, AlexNet, ResNet, SqueezeNet, and VGG16. The empirical results conclude that the CNN-LSTM based training model with ResNet architecture achieved the highest prediction accuracy of an image caption of 83%.Keywords
Funding Information
- Prince Sultan University, Riyadh, Saudi Arabia
This publication has 56 references indexed in Scilit:
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trendsJournal on Multimodal User Interfaces, 2017
- Deep Residual Learning for Image Recognition2016
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning2016
- A vision-based system to detect potholes and uneven surfaces for assisting blind people2016
- Obstacle detection and warning for visually impaired people based on electrode matrix and mobile Kinect2015
- Traffic Flow Prediction With Big Data: A Deep Learning ApproachIEEE Transactions on Intelligent Transportation Systems, 2014
- The Prevalence of Concurrent Hearing and Vision Impairment in the United StatesJAMA Internal Medicine, 2013
- Wearable Obstacle Avoidance Electronic Travel Aids for Blind: A SurveyIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2009
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998