Mixed-Query Transformer: A Unified Image Segmentation Architecture
Preprint
Abstract
Existing unified image segmentation models either employ a unified architecture across multiple tasks but use separate weights tailored to each dataset, or apply a single set of weights to multiple datasets but are limited to a single task. In this paper, we introduce the Mixed-Query Transformer (MQ-Former), a unified architecture for multi-task and multi-dataset image segmentation using a single set of weights. To enable this, we propose a mixed query strategy, which can effectively and dynamically accommodate different types of objects without heuristic designs. In addition, the unified architecture allows us to use data augmentation with synthetic masks and captions to further improve model generalization. Experiments demonstrate that MQ-Former can not only effectively handle multiple segmentation datasets and tasks compared to specialized state-of-the-art models with competitive performance, but also generalize better to open-set segmentation tasks, evidenced by over 7 points higher performance than the prior art on the open-vocabulary SeginW benchmark.Keywords
All Related Versions
This publication has 50 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- SynthCap: Augmenting Transformers with Synthetic Data for Image Captioning2023
- Revisiting Image Pyramid Structure for High Resolution Salient Object Detection2023
- Open Vocabulary Object Detection with Pseudo Bounding-Box Labels2022
- Highly Accurate Dichotomous Image Segmentation2022
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- Learning Semantic Segmentation from Multiple Datasets with Label Shifts2022
- End-to-End Object Detection with Transformers2020
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Microsoft COCO: Common Objects in Context2014