L2D2: Robot Learning from 2D drawings
Open Access
- 15 September 2025
- journal article
- research article
- Published by Springer Nature in Autonomous Robots
- Vol. 49 (3) , 1-20
Abstract
Robots should learn new tasks from humans. But how do humans convey what they want the robot to do? Existing methods largely rely on humans physically guiding the robot arm throughout their intended task. Unfortunately — as we scale up the amount of data — physical guidance becomes prohibitively burdensome. Not only do humans need to operate robot hardware but also modify the environment (e.g., moving and resetting objects) to provide multiple task examples. In this work we propose L2D2, a sketching interface and imitation learning algorithm where humans can provide demonstrations by drawing the task. L2D2 starts with a single image of the robot arm and its workspace. Using a tablet, users draw and label trajectories on this image to illustrate how the robot should act. To collect new and diverse demonstrations, we no longer need the human to physically reset the workspace; instead, L2D2 leverages vision-language segmentation to autonomously vary object locations and generate synthetic images for the human to draw upon. We recognize that drawing trajectories is not as information-rich as physically demonstrating the task. Drawings are 2-dimensional and do not capture how the robot’s actions affect its environment. To address these fundamental challenges the next stage of L2D2 grounds the human’s static, 2D drawings in our dynamic, 3D world by leveraging a small set of physical demonstrations. Our experiments and user study suggest that L2D2 enables humans to provide more demonstrations with less time and effort than traditional approaches, and users prefer drawings over physical manipulation. When compared to other drawing-based approaches, we find that L2D2 learns more performant robot policies, requires a smaller dataset, and can generalize to longer-horizon tasks. See our project website: https://collab.me.vt.edu/L2D2/Keywords
All Related Versions
Funding Information
- Division of Civil, Mechanical and Manufacturing Innovation (2205241)
This publication has 42 references indexed in Scilit:
- OpenVR: Teleoperation for manipulationSoftwareX, 2025
- View: visual imitation learning with waypointsAutonomous Robots, 2025
- Track2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation2024
- Hidden dimensions of the data: PCA vs autoencodersQuality Engineering, 2023
- Dynamic Movement Primitives Based Robot Skills LearningMachine Intelligence Research, 2023
- Detecting Twenty-Thousand Classes Using Image-Level Supervision2022
- Dimension Estimation Using Autoencoders and Application2021
- Principal component analysis: a review and recent developmentsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2016
- A survey of robot learning from demonstrationRobotics and Autonomous Systems, 2009
- Multiple View Geometry in Computer Vision2004