Anomaly Matters: An Anomaly-Oriented Model for Medical Visual Question Answering
- 29 June 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 41 (11) , 3385-3397
Abstract
Medical images contain various abnormal regions, most of which are closely related to the lesions or diseases. The abnormality or lesion is one of the major concerns during clinical practice and therefore becomes the key in answering questions about medical images. However, the recent efforts still focus on constructing a generic Visual Question Answering framework for medical-domain tasks, which is not adequate for practical medical requirements and applications. In this paper, we present two novel medical-specific modules named multiplication anomaly sensitive module and residual anomaly sensitive module to utilize weakly supervised anomaly localization information in medical Visual Question Answering. Firstly, the proposed multiplication anomaly sensitive module designed for anomaly-related questions can mask the feature of the whole image according to the anomaly location map. Secondly, the residual anomaly sensitive module could learn a flexible anomaly feature while preserving the information of the original questioned image, which is more helpful in answering anomaly-unrelated questions. Thirdly, the transformer decoder and multi-task learning strategy are combined to further enhance the question-reasoning ability and the model generalization performance. Finally, qualitative and quantitative experiments on a variety of medical datasets exhibit the superiority of the proposed approaches compared to the state-of-the-art methods.Keywords
Funding Information
- Beijing Natural Science Foundation (L192032)
- National Key Research and Development Program of China (2019YFB1406500)
- Innovation and Transformation Fund of Peking University Third Hospital (BYSYZHKC2021115)
This publication has 23 references indexed in Scilit:
- Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-Scale Lesion Database2018
- ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases2017
- Generative adversarial networks for brain lesion detection2017
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- VQA: Visual Question Answering2015
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)IEEE Transactions on Medical Imaging, 2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Glove: Global Vectors for Word Representation2014
- Multitask Learning1998
- Long Short-Term MemoryNeural Computation, 1997