RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
Preprint
Abstract
A key challenge in robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent research in one-shot imitation learning has shown promise in transferring trained policies to new tasks based on demonstrations. This feature is attractive for enabling robots to acquire new skills and improving task and motion planning. However, due to limitations in the training dataset, the current focus of the community has mainly been on simple cases, such as push or pick-place tasks, relying solely on visual guidance. In reality, there are many complex skills, some of which may even require both visual and tactile perception to solve. This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception. To achieve this, we have collected a dataset comprising over 110,000 contact-rich robot manipulation sequences across diverse skills, contexts, robots, and camera viewpoints, all collected in the real world. Each sequence in the dataset includes visual, force, audio, and action information. Moreover, we also provide a corresponding human demonstration video and a language description for each robot sequence. We have invested significant efforts in calibrating all the sensors and ensuring a high-quality dataset. The dataset is made publicly available at rh20t.github.ioKeywords
All Related Versions
This publication has 10 references indexed in Scilit:
- RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking2024
- MOMA-Force: Visual-Force Imitation for Real-World Mobile Manipulation2023
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware2023
- Self-Attention Based Visual-Tactile Fusion Learning for Predicting Grasp OutcomesIEEE Robotics and Automation Letters, 2020
- On Robustness of Multi-Modal Fusion—Robotics PerspectiveElectronics, 2020
- Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich TasksIEEE Transactions on Robotics, 2020
- Learning One-Shot Imitation From Humans Without HumansIEEE Robotics and Automation Letters, 2020
- RLBench: The Robot Learning Benchmark & Learning EnvironmentIEEE Robotics and Automation Letters, 2020
- Manipulation Skill Acquisition for Robotic Assembly Based on Multi-Modal Information DescriptionIEEE Access, 2019
- See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusionScience Robotics, 2019