Integrating Pose Features and Cross-Relationship Learning for Human–Object Interaction Detection
Open Access
Abstract
Background: The main challenge in human–object interaction detection (HOI) is how to accurately reason about ambiguous, complex, and difficult to recognize interactions. The model structure of the existing methods is relatively single, and the image input may be occluded and cannot be accurately recognized. Methods: In this paper, we design a Pose-Aware Interaction Network (PAIN) based on transformer architecture and human posture to address these issues through two innovations: A new feature fusion method is proposed, which fuses human pose features and image features early before the encoder to improve the feature expression ability, and the individual motion-related features are additionally strengthened by adding to the human branch; the Cross-Attention Relationship fusion Module (CARM) better fuses the three-branch output and captures the detailed relationship information of HOI. Results: The proposed method achieves 64.51%, 66.42% on the public dataset V-COCO and 30.83% AP on HICO-DET, which can recognize HOI instances more accurately.
Keywords
Funding Information
- National Key R&D Program of China (2023YFC3304604)
This publication has 33 references indexed in Scilit:
- HierGAT: hierarchical spatial-temporal network with graph and transformer for video HOI detectionMultimedia Systems, 2024
- Auxiliary Feature Fusion and Noise Suppression for HOI DetectionACM Transactions on Multimedia Computing, Communications, and Applications, 2024
- UAHOI: Uncertainty-aware robust interaction learning for HOI detectionComputer Vision and Image Understanding, 2024
- FGAHOI: Fine-Grained Anchors for Human-Object Interaction DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- HODN: Disentangling Human-Object Feature for HOI DetectionIEEE Transactions on Multimedia, 2023
- Transferable Interactiveness Knowledge for Human-Object Interaction DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- A novel framework and concept-based semantic search Interface for abnormal crowd behaviour analysis in surveillance videosMultimedia Tools and Applications, 2020
- OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity FieldsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Real time security framework for detecting abnormal events at ATM installationsJournal of Real-Time Image Processing, 2016
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955