Long-Term Social Interaction Context: The Key to Egocentric Addressee Detection
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8250-8254
Abstract
As embodied agents learn to interact, it is crucial for them to understand when, what, and to whom they should respond. While advances in natrual language processing and speech technologies have enabled conversational agents to focus on what to respond, they still struggle to determine when and to whom they should respond. In this paper, we address the addressee detection (Talking-To-Me, TTM) problem under the egocentric view. Instead of relying solely on short-term audio and video data, we propose a simple architecture SICNet with self/cross-modality attention that leverages long-term social interaction context. By leveraging long-term information, our approach has achieved a mean Average Precision (mAP) of 68.98% on the Ego4D TTM task, surpassing the previous state-of-the-art single-task model by 10.07%. We also conducted a detailed ablation study to demonstrate the effectiveness of each component in the long-term social interaction context.Keywords
This publication has 14 references indexed in Scilit:
- Egocentric Video Task Translation2023
- Ego4D: Around the World in 3,000 Hours of Egocentric Video2022
- MagFace: A Universal Representation for Face Recognition and Quality Assessment2021
- Are You Addressing Me? Multimodal Addressee Detection in Human-Human-Computer Conversations2017
- With whom do I interact? Detecting social interactions in egocentric photo-streams2016
- Deep Residual Learning for Image Recognition2016
- Going Deeper into First-Person Activity Recognition2016
- Bridging computer vision and social science: A multi-camera vision system for social interaction training analysis2015
- Multimodal addressee detection in multiparty dialogue systems2015
- Social interactions: A first-person perspective2012