Semi-CNN Architecture for Effective Spatio-Temporal Learning in Action Recognition
Open Access
- 12 January 2020
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 10 (2) , 557
Abstract
This paper introduces a fusion convolutional architecture for efficient learning of spatio-temporal features in video action recognition. Unlike 2D convolutional neural networks (CNNs), 3D CNNs can be applied directly on consecutive frames to extract spatio-temporal features. The aim of this work is to fuse the convolution layers from 2D and 3D CNNs to allow temporal encoding with fewer parameters than 3D CNNs. We adopt transfer learning from pre-trained 2D CNNs for spatial extraction, followed by temporal encoding, before connecting to 3D convolution layers at the top of the architecture. We construct our fusion architecture, semi-CNN, based on three popular models: VGG-16, ResNets and DenseNets, and compare the performance with their corresponding 3D models. Our empirical results evaluated on the action recognition dataset UCF-101 demonstrate that our fusion of 1D, 2D and 3D convolutions outperforms its 3D model of the same depth, with fewer parameters and reduces overfitting. Our semi-CNN architecture achieved an average of 16–30% boost in the top-1 accuracy when evaluated on an input video of 16 frames.Keywords
All Related Versions
Funding Information
- Universitetet i Tromsø (University Funding)
This publication has 17 references indexed in Scilit:
- Multi-stream CNN: Learning representations based on human-related regions for action recognitionPattern Recognition, 2018
- Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition2017
- Densely Connected Convolutional Networks2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Long-Term Temporal Convolutions for Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Finding action tubes2015
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998