TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding
Preprint
Abstract
Holistic scene understanding includes semantic segmentation, surface normal estimation, object boundary detection, depth estimation, etc. The key aspect of this problem is to learn representation effectively, as each subtask builds upon not only correlated but also distinct attributes. Inspired by visual-prompt tuning, we propose a Task-Specific Prompts Transformer, dubbed TSP-Transformer, for holistic scene understanding. It features a vanilla transformer in the early stage and tasks-specific prompts transformer encoder in the lateral stage, where tasks-specific prompts are augmented. By doing so, the transformer layer learns the generic information from the shared parts and is endowed with task-specific capacity. First, the tasks-specific prompts serve as induced priors for each task effectively. Moreover, the task-specific prompts can be seen as switches to favor task-specific representation learning for different tasks. Extensive experiments on NYUD-v2 and PASCAL-Context show that our method achieves state-of-the-art performance, validating the effectiveness of our method for holistic scene understanding. We also provide our code in the following link https://github.com/tb2-sy/TSP-Transformer.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Universal Representations: A Unified Look at Multiple Task and Domain LearningInternational Journal of Computer Vision, 2023
- All in Tokens: Unifying Output Space of Visual Tasks via Soft Token2023
- Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving2023
- Inverted Pyramid Multi-task Transformer for Dense Scene Understanding2022
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks2022
- Multi-Task Learning for Dense Prediction Tasks: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019