Improving Medical Speech-to-Text Accuracy using Vision-Language Pre-training Models
- 22 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Biomedical and Health Informatics
- Vol. 28 (3) , 1692-1703
Abstract
Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.Keywords
All Related Versions
Funding Information
- National Research Foundation of Korea (NRF-2020R1A2B5B03001980)
- National Research Foundation of Korea (RS-2023-00262527)
This publication has 31 references indexed in Scilit:
- Evaluating progress in automatic chest X-ray radiology report generationPatterns, 2023
- Multi-Modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-TrainingIEEE Journal of Biomedical and Health Informatics, 2022
- Weak-Attention Suppression for Transformer Based Speech Recognition2020
- ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context2020
- Conformer: Convolution-augmented Transformer for Speech Recognition2020
- Jasper: An End-to-End Convolutional Neural Acoustic Model2019
- wav2vec: Unsupervised Pre-Training for Speech Recognition2019
- Convolutional Neural Networks for Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech RecognitionIEEE Transactions on Audio, Speech, and Language Processing, 2011