Attention Transfer from Web Images for Video Recognition
Authors:
- Junnan LiJLJunnan Li
- Yongkang WongYWYongkang Wong
- Qi ZhaoQZQi Zhao
- Mohan KankanhalliMKMohan Kankanhalli
Preprint
Abstract
Training deep learning based video classifiers for action recognition requires a large amount of labeled videos. The labeling process is labor-intensive and time-consuming. On the other hand, large amount of weakly-labeled images are uploaded to the Internet by users everyday. To harness the rich and highly diverse set of Web images, a scalable approach is to crawl these images to train deep learning based classifier, such as Convolutional Neural Networks (CNN). However, due to the domain shift problem, the performance of Web images trained deep classifiers tend to degrade when directly deployed to videos. One way to address this problem is to fine-tune the trained models on videos, but sufficient amount of annotated videos are still required. In this work, we propose a novel approach to transfer knowledge from image domain to video domain. The proposed method can adapt to the target domain (i.e. video data) with limited amount of training data. Our method maps the video frames into a low-dimensional feature space using the class-discriminative spatial attention map for CNNs. We design a novel Siamese EnergyNet structure to learn energy functions on the attention maps by jointly optimizing two loss functions, such that the attention map corresponding to a ground truth concept would have higher energy. We conduct extensive experiments on two challenging video recognition datasets (i.e. TVHI and UCF101), and demonstrate the efficacy of our proposed method.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Do less and achieve more: Training CNNs for action recognition utilizing action images from the WebPattern Recognition, 2017
- The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition2016
- Webly-Supervised Video Recognition by Mutually Voting for Relevant Web Images and Web Video Frames2016
- Learning Visual Features from Large Weakly Supervised Data2016
- Improving Human Action Recognition Using Score Distribution and Ranking2015
- Transfer Joint Matching for Unsupervised Domain Adaptation2014
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts2014
- Subspace Interpolation via Dictionary Learning for Unsupervised Domain Adaptation2013
- Geodesic flow kernel for unsupervised domain adaptation2012