Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9485-9494
Abstract
Panoptic narrative grounding (PNG), whose core target is fine-grained image-text alignment, requires a panoptic segmentation of referred objects given a narrative caption. Previous discriminative methods achieve only weak or coarse-grained alignment by panoptic segmentation pretraining or CLIP model adaptation. Given the recent progress of text-to-image Diffusion models, several works have shown their capability to achieve fine-grained image-text alignment through cross-attention maps and improved general segmentation performance. However, the direct use of phrase features as static prompts to apply frozen Diffusion models to the PNG task still suffers from a large task gap and insufficient vision-language interaction, yielding inferior performance. Therefore, we propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features and inject the multimodal cues back, which leverages the fine-grained image-text alignment capability of Diffusion models more sufficiently. In addition, we also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement. Extensive experiments on the PNG benchmark show that our method achieves new state-of-the-art performance.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2021YFB1714300)
- Beihang World TOP University Cooperation Program
- Beijing Natural Science Foundation (L231011)
- National Science and Technology Major Project (2022ZD0115502)
- National Natural Science Foundation of China (62122010, U23B2010, 62132001)
- Zhejiang Provincial Natural Science Foundation of China (LDT23F02022F02)
- Meituan
This publication has 31 references indexed in Scilit:
- Active Perception for Visual-Language NavigationInternational Journal of Computer Vision, 2022
- Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation2021
- Linguistic Structure Guided Context Modeling for Referring Image Segmentation2020
- Connecting Vision and Language with Localized Narratives2020
- Referring Image Segmentation via Cross-Modal Progressive Comprehension2020
- Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations2017
- Segmentation from Natural Language Expressions2016
- Deep Residual Learning for Image Recognition2016
- Fully convolutional networks for semantic segmentation2015
- Microsoft COCO: Common Objects in Context2014