Jointly-Learnt Networks for Future Action Anticipation via Self-Knowledge Distillation and Cycle Consistency
- 23 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (7) , 3243-3256
Abstract
Future action anticipation aims to infer future actions from the observation of a small set of past video frames. In this paper, we propose a novel Jointly-learnt Action Anticipation Network (J-AAN) via Self-Knowledge Distillation (Self-KD) and cycle consistency for future action anticipation. In contrast to the current state-of-the-art methods which anticipate the future actions either directly or recursively, our proposed J-AAN anticipates the future actions jointly in both direct and recursive ways. However, when dealing with future action anticipation, one important challenge to address is the future’s uncertainty since multiple action sequences may come from or be followed by the same action. Training an action anticipation model with one-hot-encoded hard labels that assign zero probabilities to incorrect yet semantically similar actions may not handle the uncertain future. To address this challenge, we design a Self-KD mechanism to train our J-AAN, where the J-AAN gradually distills its own knowledge during the training to soften the hard labels to model the uncertainty on future action anticipation. Furthermore, we design a forward and backward action anticipation framework with our proposed J-AAN based on a cyclic consistency constraint. The forward J-AAN anticipates the future actions from the observed past actions, and the backward J-AAN verifies the anticipation of the forward J-AAN by anticipating the past actions from the anticipated future actions. The proposed method outperforms all the latest state-of-the-art action anticipation methods on the Breakfast, 50Salads, and EPIC-Kitchens-55 datasets. This project will be publicly available on https://github.com/MoniruzzamanMd/J-AAN.Keywords
Funding Information
- National Science Foundation via Cyber-Physical Systems (CMMI-1646162)
- National Robotics Initiative (CMMI-1954548)
- Human-Technology Frontier (ECCS-2025929)
This publication has 69 references indexed in Scilit:
- Human Interaction Prediction Using Deep Temporal Features2016
- Multi-region Two-Stream R-CNN for Action Detection2016
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Fast Action Localization in Large-Scale Video ArchivesIEEE Transactions on Circuits and Systems for Video Technology, 2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities2014
- 3D Convolutional Neural Networks for Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- Back-Translation for Cross-Cultural ResearchJournal of Cross-Cultural Psychology, 1970