Video Recognition in Portrait Mode
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 21831-21841
Abstract
The creation of new datasets often presents new challenges for video recognition and can inspire novel ideas while addressing these challenges. While existing datasets mainly comprise landscape mode videos, our paper seeks to introduce portrait mode videos to the research community and highlight the unique challenges associated with this video format. With the growing popularity of smartphones and social media applications, recognizing portrait mode videos is becoming increasingly important. To this end, we have developed the first dataset dedicated to portrait mode video recognition, namely PortraitMode-400. The taxon-omy of PortraitMode-400 was constructed in a data-driven manner, comprising 400 fine-grained categories, and rigorous quality assurance was implemented to ensure the accuracy of human annotations. In addition to the new dataset, we conducted a comprehensive analysis of the impact of video format (portrait mode versus landscape mode) on recognition accuracy and spatial bias due to the different formats. Furthermore, we designed extensive experiments to explore key aspects of portrait mode video recognition, including the choice of data augmentation, evaluation procedure, the importance of temporal information, and the role of audio modality. Building on the insights from our experimental results and the introduction of PortraitMode-400, our paper aims to inspire further research efforts in this emerging research direction.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- Videos as Space-Time Region Graphs2018
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Bag of visual words and fusion methods for action recognition: Comprehensive study and good practiceComputer Vision and Image Understanding, 2016
- A survey on still image based human action recognitionPattern Recognition, 2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Dense Trajectories and Motion Boundary Descriptors for Action RecognitionInternational Journal of Computer Vision, 2013
- Fast unfolding of communities in large networksJournal of Statistical Mechanics: Theory and Experiment, 2008
- Learning realistic human actions from movies2008
- Human Activity Recognition with Metric Learning2008
- On Space-Time Interest PointsInternational Journal of Computer Vision, 2005