Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding
- 27 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (2) , 1181-1198
Abstract
Multimodal transformer exhibits high capacity and flexibility to align image and text for visual grounding. However, the existing encoder-only grounding framework (e.g., TransVG) suffers from heavy computation due to the self-attention operation with quadratic time complexity. To address this issue, we present a new multimodal transformer architecture, coined as Dynamic M utilmodal detection transformer ( DETR ) (Dynamic MDETR), by decoupling the whole grounding process into encoding and decoding phases. The key observation is that there exists high spatial redundancy in images. Thus, we devise a new dynamic multimodal transformer decoder by exploiting this sparsity prior to speed up the visual grounding process. Specifically, our dynamic decoder is composed of a 2D adaptive sampling module and a text guided decoding module. The sampling module aims to select these informative patches by predicting the offsets with respect to a reference point, while the decoding module works for extracting the grounded object information by performing cross attention between image features and text features. These two modules are stacked alternatively to gradually bridge the modality gap and iteratively refine the reference point of grounded object, eventually realizing the objective of visual grounding. Extensive experiments on five benchmarks demonstrate that our proposed Dynamic MDETR achieves competitive trade-offs between computation and accuracy. Notably, using only 9% feature points in the decoder, we can reduce $\sim$ 44% GFLOPs of the multimodal transformer, but still get higher accuracy than the encoder-only counterpart. With the same number of encoder layers as TransVG, our Dynamic MDETR (ResNet-50) outperforms TransVG (ResNet-101) but only brings marginal extra computational cost relative to TransVG. In addition, to verify its generalization ability and scale up our Dynamic MDETR, we build the first one-stage CLIP empowered visual grounding framework, and achieve the state-of-the-art performance on these benchmarks.
Keywords
All Related Versions
Funding Information
- National Key R & D Program of China (2022ZD0160900)
- National Natural Science Foundation of China (62076119, 61921006)
- Fundamental Research Funds for the Central Universities (020214380091)
- Collaborative Innovation Center of Novel Software Technology and Industrialization
This publication has 45 references indexed in Scilit:
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Modeling Relationships in Referential Expressions with Compositional Modular Networks2017
- Modeling Context in Referring Expressions2016
- Modeling Context Between Objects for Referring Expression Understanding2016
- Deep Residual Learning for Image Recognition2016
- Natural Language Object Retrieval2016
- Generation and Comprehension of Unambiguous Object Descriptions2016
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- Microsoft COCO: Common Objects in Context2014
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014