Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Models
Authors:
- Haonan ChenHCHaonan Chen
- Jiaming XuJXJiaming Xu
- Lily ShengLSLily Sheng
- Tianchen JiTJTianchen Ji
- Shuijing LiuSLShuijing Liu
- Yunzhu LiYunzhu Li
- Katherine Driggs-CampbellKDKatherine Driggs-Campbell
Preprint
Abstract
When performing tasks like laundry, humans naturally coordinate both hands to manipulate objects and anticipate how their actions will change the state of the clothes. However, achieving such coordination in robotics remains challenging due to the need to model object movement, predict future states, and generate precise bimanual actions. In this work, we address these challenges by infusing the predictive nature of human manipulation strategies into robot imitation learning. Specifically, we disentangle task-related state transitions from agent-specific inverse dynamics modeling to enable effective bimanual coordination. Using a demonstration dataset, we train a diffusion model to predict future states given historical observations, envisioning how the scene evolves. Then, we use an inverse dynamics model to compute robot actions that achieve the predicted states. Our key insight is that modeling object movement can help learning policies for bimanual coordination manipulation tasks. Evaluating our framework across diverse simulation and real-world manipulation setups, including multimodal goal configurations, bimanual manipulation, deformable objects, and multi-object setups, we find that it consistently outperforms state-of-the-art state-to-action mapping policies. Our method demonstrates a remarkable capacity to navigate multimodal goal configurations and action distributions, maintain stability across different control modes, and synthesize a broader range of behaviors than those present in the demonstration dataset.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators2024
- Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots2024
- RoboCraft: Learning to see, simulate, and shape elasto-plastic objects in 3D with graph networksThe International Journal of Robotics Research, 2023
- Goal-Conditioned Imitation Learning using Score-based Diffusion Policies2023
- Augmenting physical models with deep networks for complex dynamics forecasting*Journal of Statistical Mechanics: Theory and Experiment, 2021
- Action prediction during real‐time parent‐infant interactionsDevelopmental Science, 2020
- Pushing revisited: Differential flatness, trajectory planning, and stabilizationThe International Journal of Robotics Research, 2019
- Trends and challenges in robot manipulationScience, 2019
- Infants anticipate probabilistic but not deterministic outcomesCognition, 2016
- Learning the dynamics of doors for robotic manipulation2013