Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
- 21 September 2021
- book chapter
- Published by Springer Nature
- Vol. 12966, 110-119
Abstract
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR. Our code is available at https://github.com/mshaikh2/JoImTeR_MLMI_2021. Keywords Self-supervised learning Multi-modality AttentionKeywords
All Related Versions
This publication has 21 references indexed in Scilit:
- TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays2018
- ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases2017
- Deep Visual-Semantic Alignments for Generating Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Stacked Attention Networks for Image Question Answering2016
- Preparing a collection of radiology examinations for distribution and retrievalJournal of the American Medical Informatics Association, 2015
- Deep visual-semantic alignments for generating image descriptions2015
- From captions to visual concepts and back2015
- Convolutional Neural Networks for Sentence Classification2014
- Learning deep structured semantic models for web search using clickthrough data2013