Few-Shot Human-Object Interaction Recognition With Semantic-Guided Attentive Prototypes Network
- 31 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 1648-1661
Abstract
Extreme instance imbalance among categories and combinatorial explosion make the recognition of Human-Object Interaction (HOI) a challenging task. Few studies have addressed both challenges directly. Motivated by the success of few-shot learning that learns a robust model from a few instances, we formulate HOI as a few-shot task in a meta-learning framework to alleviate the above challenges. Due to the fact that the intrinsical characteristic of HOI is diverse and interactive, we propose a Semantic-guided Attentive Prototypes Network (SAPNet) framework to learn a semantic-guided metric space where HOI recognition can be performed by computing distances to attentive prototypes of each class. Specifically, the model generates attentive prototypes guided by the category names of actions and objects, which highlight the commonalities of images from the same class in HOI. In addition, we design two alternative prototypes calculation methods, i.e., Prototypes Shift (PS) approach and Hallucinatory Graph Prototypes (HGP) approach, which explore to learn a suitable category prototypes representations in HOI. Finally, in order to realize the task of few-shot HOI, we reorganize 2 HOI benchmark datasets with 2 split strategies, i.e., HICO-NN, TUHOI-NN, HICO-NF, and TUHOI-NF. Extensive experimental results on these datasets have demonstrated the effectiveness of our proposed SAPNet approach.Keywords
Funding Information
- National Natural Science Foundation of China (61771329, 61632018)
- Natural Science Foundation of Tianjin (19JCYBJC16000)
This publication has 27 references indexed in Scilit:
- Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach2017
- Residual Attention Network for Image Classification2017
- Deep Residual Learning for Image Recognition2016
- Training Region-Based Object Detectors with Online Hard Example Mining2016
- HICO: A Benchmark for Recognizing Human-Object Interactions in Images2015
- Contextual Action Recognition with R* CNN2015
- FaceNet: A unified embedding for face recognition and clustering2015
- Main objects interaction activity recognition in real imagesNeural Computing and Applications, 2015
- TUHOI: Trento Universal Human Object Interaction Dataset2014
- Objects in Action: An Approach for Combining Action Understanding and Object Perception2007