Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11210-11219
Abstract
Modern one-stage video instance segmentation networks suffer from two limitations. First, convolutional features are neither aligned with anchor boxes nor with ground-truth bounding boxes, reducing the mask sensitivity to spatial location. Second, a video is directly divided into individual frames for frame-level instance segmentation, ignoring the temporal correlation between adjacent frames. To address these issues, we propose a simple yet effective one-stage video instance segmentation framework by spatial calibration and temporal fusion, namely STMask. To ensure spatial feature calibration with ground-truth bounding boxes, we first predict regressed bounding boxes around ground-truth bounding boxes, and extract features from them for frame-level instance segmentation. To further explore temporal correlation among video frames, we aggregate a temporal fusion module to infer instance masks from each frame to its adjacent frames, which helps our frame-work to handle challenging videos such as motion blur, partial occlusion and unusual object-to-camera poses. Experiments on the YouTube-VIS valid set show that the proposed STMask with ResNet-50/-101 backbone obtains 33.5 % / 36.8 % mask AP, while achieving 28.6 / 23.4 FPS on video instance segmentation. The code is released online https://github.com/MinghanLi/STMask.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Fully Motion-Aware Network for Video Object Detection2018
- Box2Pix: Single-Shot Instance Segmentation by Assigning Pixels to Object Boxes2018
- Proposal-Free Network for Instance-Level Object SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Detect to Track and Track to Detect2017
- Flow-Guided Feature Aggregation for Video Object Detection2017
- Mask R-CNN2017
- Fully Convolutional Instance-Aware Semantic Segmentation2017
- InstanceCut: From Edges to Instances with MultiCut2017
- Instance-Aware Semantic Segmentation via Multi-task Network Cascades2016
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014