KiUT: Knowledge-injected U-Transformer for Radiology Report Generation
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19809-19818
Abstract
Radiology report generation aims to automatically generate a clinically accurate and coherent paragraph from the X-ray image, which could relieve radiologists from the heavy burden of report writing. Although various image caption methods have shown remarkable performance in the natural image field, generating accurate reports for medical images requires knowledge of multiple modalities, including vision, language, and medical terminology. We propose a Knowledge-injected U-Transformer (KiUT) to learn multi-level visual representation and adaptively distill the information with contextual and clinical knowledge for word prediction. In detail, a U-connection schema between the encoder and decoder is designed to model interactions between different modalities. And a symptom graph and an injected knowledge distiller are developed to assist the report generation. Experimentally, we outperform state-of-the-art methods on two widely used benchmark datasets: IU-Xray and MIMIC-CXR. Further experimental results prove the advantages of our architecture and the complementary benefits of the injected knowledge.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Multimodal Recurrent Model with Attention for Automated Radiology Report Generation2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Recurrent Topic-Transition GAN for Visual Paragraph Generation2017
- Densely Connected Convolutional Networks2017
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Self-Critical Sequence Training for Image Captioning2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning ChallengeIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015