METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11558-11567
Abstract
In clinical scenarios, multi-specialist consultation could significantly benefit the diagnosis, especially for intricate cases. This inspires us to explore a “multi-expert joint diagnosis” mechanism to upgrade the existing “single expert” framework commonly seen in the current literature. To this end, we propose METransformer, a method to realize this idea with a transformer-based backbone. The key design of our method is the introduction of multiple learnable “expert” tokens into both the transformer encoder and decoder. In the encoder, each expert token interacts with both vision tokens and other expert tokens to learn to attend different image regions for image representation. These expert tokens are encouraged to capture complementary information by an orthogonal loss that minimizes their over-lap. In the decoder, each attended expert token guides the cross-attention between input words and visual tokens, thus influencing the generated report. A metrics-based expert voting strategy is further developed to generate the final report. By the multi-experts concept, our model enjoys the merits of an ensemble-based approach but through a manner that is computationally more efficient and supports more sophisticated interactions among experts. Experimental results demonstrate the promising performance of our proposed model on two widely used benchmarks. Last but not least, the framework-level innovation makes our work ready to incorporate advances on existing “single-expert” models to further improve its performance.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Improved Disease Classification in Chest X-Rays with Transferred Features from Report Generation2019
- Squeeze-and-Excitation Networks2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- On the Automatic Generation of Medical Imaging Reports2018
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Self-Critical Sequence Training for Image Captioning2017
- Image Captioning with Semantic Attention2016
- Bilinear CNN Models for Fine-Grained Visual Recognition2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015