Spatiotemporal Fusion in 3D CNNs: A Probabilistic View
Preprint
Abstract
Despite the success in still image recognition, deep neural networks for spatiotemporal signal tasks (such as human action recognition in videos) still suffers from low efficacy and inefficiency over the past years. Recently, human experts have put more efforts into analyzing the importance of different components in 3D convolutional neural networks (3D CNNs) to design more powerful spatiotemporal learning backbones. Among many others, spatiotemporal fusion is one of the essentials. It controls how spatial and temporal signals are extracted at each layer during inference. Previous attempts usually start by ad-hoc designs that empirically combine certain convolutions and then draw conclusions based on the performance obtained by training the corresponding networks. These methods only support network-level analysis on limited number of fusion strategies. In this paper, we propose to convert the spatiotemporal fusion strategies into a probability space, which allows us to perform network-level evaluations of various fusion strategies without having to train them separately. Besides, we can also obtain fine-grained numerical information such as layer-level preference on spatiotemporal fusion within the probability space. Our approach greatly boosts the efficiency of analyzing spatiotemporal fusion. Based on the probability space, we further generate new fusion strategies which achieve the state-of-the-art performance on four well-known action recognition datasets.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Single Path One-Shot Neural Architecture Search with Uniform Sampling2020
- SlowFast Networks for Video Recognition2019
- Dense 3D-Convolutional Neural Network for Person Re-Identification in VideosACM Transactions on Multimedia Computing, Communications, and Applications, 2019
- Learning Compact Appearance Representation for Video-Based Person Re-IdentificationIEEE Transactions on Circuits and Systems for Video Technology, 2018
- The “Something Something” Video Database for Learning and Evaluating Visual Common Sense2017
- Long-Term Temporal Convolutions for Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017