Medical visual question answering using joint self-supervised learning
Preprint
Abstract
Visual Question Answering (VQA) becomes one of the most active research problems in the medical imaging domain. A well-known VQA challenge is the intrinsic diversity between the image and text modalities, and in the medical VQA task, there is another critical problem relying on the limited size of labelled image-question-answer data. In this study we propose an encoder-decoder framework that leverages the image-text joint representation learned from large-scaled medical image-caption data and adapted to the small-sized medical VQA task. The encoder embeds across the image-text dual modalities with self-attention mechanism and is independently pre-trained on the large-scaled medical image-caption dataset by multiple self-supervised learning tasks. Then the decoder is connected to the top of the encoder and fine-tuned using the small-sized medical VQA dataset. The experiment results present that our proposed method achieves better performance comparing with the baseline and SOTA methods.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Deep Modular Co-Attention Networks for Visual Question Answering2019
- A dataset of clinically generated visual questions and answers about radiology imagesScientific Data, 2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays2018
- MUTAN: Multimodal Tucker Fusion for Visual Question Answering2017
- VQA: Visual Question AnsweringInternational Journal of Computer Vision, 2016
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Stacked Attention Networks for Image Question Answering2016
- Microsoft COCO: Common Objects in Context2014