Streaming Video Model
Preprint
Abstract
Video understanding tasks have traditionally been modeled by two separate architectures, specially tailored for two distinct tasks. Sequence-based video tasks, such as action recognition, use a video backbone to directly extract spatiotemporal features, while frame-based video tasks, such as multiple object tracking (MOT), rely on single fixed-image backbone to extract spatial features. In contrast, we propose to unify video understanding tasks into one novel streaming video architecture, referred to as Streaming Vision Transformer (S-ViT). S-ViT first produces frame-level features with a memory-enabled temporally-aware spatial encoder to serve the frame-based video tasks. Then the frame features are input into a task-related temporal decoder to obtain spatiotemporal features for sequence-based tasks. The efficiency and efficacy of S-ViT is demonstrated by the state-of-the-art accuracy in the sequence-based action recognition task and the competitive advantage over conventional architecture in the frame-based MOT task. We believe that the concept of streaming video model and the implementation of S-ViT are solid steps towards a unified deep learning architecture for video understanding. Code will be available at https://github.com/yuzhms/Streaming-Video-Model.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Frozen CLIP Models are Efficient Video Learners2022
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- FairMOT: On the Fairness of Detection and Re-identification in Multiple Object TrackingInternational Journal of Computer Vision, 2021
- Quasi-Dense Similarity Learning for Multiple Object Tracking2021
- X3D: Expanding Architectures for Efficient Video Recognition2020
- Augment Your Batch: Improving Generalization Through Instance Repetition2020
- Path Aggregation Network for Instance Segmentation2018
- A Closer Look at Spatiotemporal Convolutions for Action Recognition2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Deep Residual Learning for Image Recognition2016