Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision
Authors:
- Yanwei LiYLYanwei Li
- Hengshuang ZhaoHZHengshuang Zhao
- Xiaojuan QiXQXiaojuan Qi
- Yukang ChenYCYukang Chen
- Lu QiLQLu Qi
- Liwei WangLWLiwei Wang
- Zeming LiZLZeming Li
- Jian SunJSJian Sun
- Jiaya JiaJJJiaya Jia
Preprint
Abstract
In this paper, we present a conceptually simple, strong, and efficient framework for fully- and weakly-supervised panoptic segmentation, called Panoptic FCN. Our approach aims to represent and predict foreground things and background stuff in a unified fully convolutional pipeline, which can be optimized with point-based fully or weak supervision. In particular, Panoptic FCN encodes each object instance or stuff category with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly. With this approach, instance-aware and semantically consistent properties for things and stuff can be respectively satisfied in a simple generate-kernel-then-segment workflow. Without extra boxes for localization or instance separation, the proposed approach outperforms the previous box-based and -free models with high efficiency. Furthermore, we propose a new form of point-based annotation for weakly-supervised panoptic segmentation. It only needs several random points for both things and stuff, which dramatically reduces the annotation cost of human. The proposed Panoptic FCN is also proved to have much superior performance in this weakly-supervised setting, which achieves 82% of the fully-supervised performance with only 20 randomly annotated points per instance. Extensive experiments demonstrate the effectiveness and efficiency of Panoptic FCN on COCO, VOC 2012, Cityscapes, and Mapillary Vistas datasets. And it sets up a new leading benchmark for both fully- and weakly-supervised panoptic segmentation. Our code and models are made publicly available at https://github.com/dvlab-research/PanopticFCN.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- GrabCut: Interactive Foreground Extraction Using Iterated Graph Cuts2023
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Learning Category- and Instance-Aware Pixel Embedding for Fast Panoptic SegmentationIEEE Transactions on Image Processing, 2021
- PointINS: Point-Based Instance SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- Towards Bounding-Box Free Panoptic Segmentation2021
- Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal GenerationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014