Learning by Watching: A Review of Video-Based Learning Approaches for Robot Manipulation
Open Access
- 23 October 2025
- journal article
- review article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 184071-184109
Abstract
Robot learning of manipulation skills is hindered by the scarcity of diverse, unbiased datasets. While curated datasets can help, challenges remain in generalizability and real-world transfer. Meanwhile, large-scale ’in-the-wild’ video datasets have driven progress in computer vision using self-supervised techniques. Translating this to robotics, recent works have explored learning manipulation skills using abundant passive videos sourced online. Showing promising results, such video-based learning paradigms provide scalable supervision while reducing dataset bias. This survey reviews foundations such as video feature representation learning techniques, object affordance understanding, 3D hand/body modeling, and large-scale robot resources, as well as emerging techniques for acquiring robot manipulation skills from uncontrolled video demonstrations.We discuss how learning from only observing large-scale human videos can enhance generalization and sample efficiency for robotic manipulation. The survey summarizes video-based learning approaches, analyzes their benefits over standard datasets, survey metrics, and benchmarks, and discusses open challenges and future directions in this nascent domain at the intersection of computer vision, natural language processing, and robot learning.Keywords
All Related Versions
This publication has 125 references indexed in Scilit:
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos2022
- Learning Collaborative Action Plans from YouTube Videos2022
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100International Journal of Computer Vision, 2021
- Learning Predictive Models from Observation and Interaction2020
- Review and Analysis of Zero, One and Few Shot Learning Approaches2019
- Multimodal Unsupervised Image-to-Image Translation2018
- Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videosPattern Recognition, 2018
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Physically Grounded Spatio-temporal Object Affordances2014