Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model
Preprint
Abstract
Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Evaluating progress in automatic chest X-ray radiology report generationPatterns, 2023
- Multi-Modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-TrainingIEEE Journal of Biomedical and Health Informatics, 2022
- Jasper: An End-to-End Convolutional Neural Acoustic Model2019
- wav2vec: Unsupervised Pre-Training for Speech Recognition2019
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Conversational speech transcription using context-dependent deep neural networks2011
- Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech RecognitionIEEE Transactions on Audio, Speech, and Language Processing, 2011
- The Application of Hidden Markov Models in Speech RecognitionFoundations and Trends® in Signal Processing, 2008
- Julius --- an open source real-time large vocabulary recognition engine2001
- A tutorial on hidden Markov models and selected applications in speech recognitionProceedings of the IEEE, 1989