Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
Authors:
- 19 October 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6342-6352
Abstract
This paper investigates the problem of understanding dynamic 3D scenes from egocentric observations, a key challenge in robotics and embodied AI. Unlike prior studies that explored this as long-form video understanding and utilized egocentric video only, we instead propose an LLMbased agent, Embodied VideoAgent, which constructs scene memory from both egocentric video and embodied sensory inputs (e.g. depth and pose sensing). We further introduce a VLM-based approach to automatically update the memory when actions or activities over objects are perceived. Embodied VideoAgent attains significant advantages over counterparts in challenging reasoning and planning tasks in 3D scenes, achieving gains of 6.5% on Ego4D-VQ3D, 2.6% on OpenEQA, and 15.3% on EnvQA. We have also demonstrated its potential in various embodied AI tasks including generating embodied interactions and perception for robot manipulation. The code and demo will be made public.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind2025
- Text-Conditioned Resampler For Long Form Video Understanding2024
- SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- Unifying 3D Vision-Language Understanding via Promptable Queries2024
- VideoAgent: A Memory-Augmented Multimodal Agent for Video Understanding2024
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100International Journal of Computer Vision, 2021
- Towards Long-Form Video Understanding2021
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions2021
- MovieQA: Understanding Stories in Movies through Question-Answering2016