Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind
Authors:
- 25 March 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1211-1221
Abstract
As humans move around, performing their daily tasks, they are able to recall where they have positioned objects in their environment, even if these objects are currently out of their sight. In this paper, we aim to mimic this spatial cognition ability. We thus formulate the task of Out of Sight, Not Out of Mind - 3D tracking active objects using observations captured through an egocentric camera. We introduce a simple but effective approach to address this challenging problem, called Lift, Match, and Keep (LMK). LMK lifts partial 2D observations to 3D world coordinates, matches them over time using visual appearance, 3D location and interactions to form object tracks, and keeps these object tracks even when they go out-of-view of the camera. We benchmark LMK on 100 long videos from EPICKITCHENS. Our results demonstrate that spatial cognition is critical for correctly locating objects over short and long time scales. E.g., for one long egocentric video, we estimate the 3D location of 50 active objects. After 120 seconds, 57 % of the objects are correctly localised by LMK, compared to just 33% by a recent 3D method for egocentric videos and 17 % by a general 2D tracking method.Keywords
All Related Versions
This publication has 51 references indexed in Scilit:
- Egocentric Activity Recognition and Localization on a 3D Map2022
- MOTR: End-to-End Multiple-Object Tracking with Transformer2022
- ByteTrack: Multi-object Tracking by Associating Every Detection Box2022
- Object Permanence2022
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100International Journal of Computer Vision, 2021
- FairMOT: On the Fairness of Detection and Re-identification in Multiple Object TrackingInternational Journal of Computer Vision, 2021
- Learning Object Permanence from Video2020
- HOTA: A Higher Order Metric for Evaluating Multi-object TrackingInternational Journal of Computer Vision, 2020
- Performance Measures and a Data Set for Multi-target, Multi-camera Tracking2016
- Pixelwise View Selection for Unstructured Multi-View Stereo2016