DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (6) , 5814-5822
Abstract
Despite the great potential in capturing long-range dependency, one rarely-explored underlying issue of transformer in medical image segmentation is attention collapse, making it often degenerate into a bypass module in CNN-Transformer hybrid architectures. This is due to the high computational complexity of vision transformers requiring extensive training data while well-annotated medical image data is relatively limited, resulting in poor convergence. In this paper, we propose a plug-n-play transformer block with dynamic token merging, named DTMFormer, to avoid building long-range dependency on redundant and duplicated tokens and thus pursue better convergence. Specifically, DTMFormer consists of an attention-guided token merging (ATM) module to adaptively cluster tokens into fewer semantic tokens based on feature and dependency similarity and a light token reconstruction module to fuse ordinary and semantic tokens. In this way, as self-attention in ATM is calculated based on fewer tokens, DTMFormer is of lower complexity and more friendly to converge. Extensive experiments on publicly-available datasets demonstrate the effectiveness of DTMFormer working as a plug-n-play module for simultaneous complexity reduction and performance improvement. We believe it will inspire future work on rethinking transformers in medical image segmentation. Code: https://github.com/iam-nacl/DTMFormer.Keywords
This publication has 34 references indexed in Scilit:
- Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation2023
- D-former: a U-shaped Dilated Transformer for 3D medical image segmentationNeural Computing and Applications, 2022
- Patcher: Patch Transformers with Mixture of Experts for Precise Medical Image Segmentation2022
- Multi-compound Transformer for Accurate Biomedical Image Segmentation2021
- TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation2021
- Medical Transformer: Gated Axial-Attention for Medical Image Segmentation2021
- Attention gated networks: Learning to leverage salient regions in medical imagesMedical Image Analysis, 2019
- UNet++: A Nested U-Net Architecture for Medical Image Segmentation2018
- Study on density peaks clustering based on k-nearest neighbors and principal component analysisKnowledge-Based Systems, 2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015