SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos
Authors:
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3430-3438
Abstract
Action anticipation in egocentric videos is a difficult task due to the inherently multi-modal nature of human actions. Additionally, some actions happen faster or slower than others depending on the actor or surrounding context which could vary each time and lead to different predictions. Based on this idea, we build upon RULSTM architecture, which is specifically designed for anticipating human actions, and propose a novel attention-based technique to evaluate, simultaneously, slow and fast features extracted from three different modalities, namely RGB, optical flow and extracted objects. Two branches process information at different time scales, i.e., frame-rates, and several fusion schemes are considered to improve prediction accuracy. We perform extensive experiments on EpicKitchens55 and EGTEA Gaze+ datasets, and demonstrate that our technique systematically improves the results of RULSTM architecture for Top-5 accuracy metric at different anticipation times.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Multi-Modal Domain Adaptation for Fine-Grained Action Recognition2020
- Symbiotic Attention with Privileged Information for Egocentric Action RecognitionProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Modality Compensation Network: Cross-Modal Adaptation for Action RecognitionIEEE Transactions on Image Processing, 2020
- Multi-Modal Multi-Scale Deep Learning for Large-Scale Image AnnotationIEEE Transactions on Image Processing, 2018
- In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video2018
- When will you do what? - Anticipating Temporal Occurrences of Activities2018
- Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction2018
- Next-active-object prediction from egocentric videosJournal of Visual Communication and Image Representation, 2017
- Anticipating Human Activities Using Object Affordances for Reactive Robotic ResponseIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Large-Scale Video Classification with Convolutional Neural Networks2014