Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations
Authors:
- Jonathan TremblayJTJonathan Tremblay
- Thang ToTTThang To
- Artem MolchanovAMArtem Molchanov
- Stephen TyreeSTStephen Tyree
- Jan KautzJKJan Kautz
- Stan BirchfieldSBStan Birchfield
Preprint
Abstract
We present a system to infer and execute a human-readable program from a real-world demonstration. The system consists of a series of neural networks to perform perception, program generation, and program execution. Leveraging convolutional pose machines, the perception network reliably detects the bounding cuboids of objects in real images even when severely occluded, after training only on synthetic images using domain randomization. To increase the applicability of the perception network to new scenarios, the network is formulated to predict in image space rather than in world space. Additional networks detect relationships between objects, generate plans, and determine actions to reproduce a real-world demonstration. The networks are trained entirely in simulation, and the system is tested in the real world on the pick-and-place problem of stacking colored cubes using a Baxter robot.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Weakly-Supervised Learning of Visual Relations2017
- Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks2017
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning2017
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017
- Weakly Supervised Dense Video Captioning2017
- Learning modular neural network policies for multi-task and multi-robot transfer2017
- Convolutional Pose Machines2016
- Show and tell: A neural image caption generator2015
- 2D Human Pose Estimation: New Benchmark and State of the Art Analysis2014
- Progressive search space reduction for human pose estimation2008