Controlling diverse robots by inferring Jacobian fields with deep networks
Open Access
- 25 June 2025
- journal article
- research article
- Published by Springer Nature in Nature
- Vol. 643 (8070) , 89-95
Abstract
Mirroring the complex structures and diverse functions of natural organisms is a long-standing challenge in robotics1–4. Modern fabrication techniques have greatly expanded the feasible hardware5–8, but using these systems requires control software to translate the desired motions into actuator commands. Conventional robots can easily be modelled as rigid links connected by joints, but it remains an open challenge to model and control biologically inspired robots that are often soft or made of several materials, lack sensing capabilities and may change their material properties with use9–12. Here, we introduce a method that uses deep neural networks to map a video stream of a robot to its visuomotor Jacobian field (the sensitivity of all 3D points to the robot’s actuators). Our method enables the control of robots from only a single camera, makes no assumptions about the robots’ materials, actuation or sensing, and is trained without expert intervention by observing the execution of random commands. We demonstrate our method on a diverse set of robot manipulators that vary in actuation, materials, fabrication and cost. Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot. Because it enables robot control using a generic camera as the only sensor, we anticipate that our work will broaden the design space of robotic systems and serve as a starting point for lowering the barrier to robotic automation.Keywords
All Related Versions
This publication has 65 references indexed in Scilit:
- Effortless creation of safe robots from modules through self-programming and self-verificationScience Robotics, 2019
- Soft robot perception using embedded soft sensors and recurrent neural networksScience Robotics, 2019
- Handedness in shearing auxetics creates rigid and compliant structuresScience, 2018
- The grand challenges of Science RoboticsScience Robotics, 2018
- Fluid-driven origami-inspired artificial musclesProceedings of the National Academy of Sciences, 2017
- Soft robotics: Technologies and systems pushing the boundaries of robot abilitiesScience Robotics, 2016
- Design, fabrication and control of soft robotsNature, 2015
- Materials that couple sensing, actuation, computation, and communicationScience, 2015
- Soft robotics: a bioinspired evolution in roboticsTrends in Biotechnology, 2013
- Self-Organization, Embodiment, and Biologically Inspired RoboticsScience, 2007