All in Tokens: Unifying Output Space of Visual Tasks via Soft Token
Open Access
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19843-19853
Abstract
We introduce AiT, a unified output representation for various vision tasks, which is a crucial step towards general-purpose vision task solvers. Despite the challenges posed by the high-dimensional and task-specific outputs, we showcase the potential of using discrete representation (VQVAE) to model the dense outputs of many computer vision tasks as a sequence of discrete tokens. This is inspired by the established ability of VQ-VAE to conserve the structures spanning multiple pixels using few discrete codes. To that end, we present a modified shallower architecture for VQ-VAE that improves efficiency while keeping prediction accuracy. Our approach also incorporates uncertainty into the decoding process by using a soft fusion of the codebook entries, providing a more stable training process, which notably improved prediction accuracy. Our evaluation of AiT on depth estimation and instance segmentation tasks, with both continuous and discrete labels, demonstrates its superiority compared to other unified models. The code and models are available at https://github.com/SwinTransformer/AiT.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- DepthFormer: Exploiting Long-range Correlation and Local Information for Accurate Monocular Depth EstimationMachine Intelligence Research, 2023
- Inverted Pyramid Multi-task Transformer for Dense Scene Understanding2022
- LocalBins: Improving Depth Estimation by Learning Local Distributions2022
- Dense RepPoints: Representing Visual Objects with Dense Point Sets2020
- End-to-End Object Detection with Transformers2020
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Simple Baselines for Human Pose Estimation and Tracking2018
- Unified Perceptual Parsing for Scene Understanding2018
- Deep convolutional neural fields for depth estimation from a single image2015
- Indoor Segmentation and Support Inference from RGBD Images2012