DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability
Authors:
- 14 October 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1412-1419
Abstract
Generative models such as diffusion models, excel at capturing high-dimensional distributions with diverse input modalities, e.g. robot trajectories, but are less effective at multistep constraint reasoning. Task and Motion Planning (TAMP) approaches are suited for planning multi-step autonomous robot manipulation. However, it can be difficult to apply them to domains where the environment and its dynamics are not fully known. We propose to overcome these limitations by composing diffusion models using a TAMP system. We use the learned components for constraints and samplers that are difficult to engineer in the planning model, and use a TAMP solver to search for the task plan with constraint-satisfying action parameter values. To tractably make predictions for unseen objects in the environment, we define the learned samplers and TAMP operators on learned latent embedding of changing object states. We evaluate our approach in a simulated articulated object manipulation domain and show how the combination of classical TAMP, generative modeling, and latent embedding enables multi-step constraint-based reasoning. We also apply the learned sampler in the real world. Website: https://sites.google.com/view/dimsam-tamp.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models2023
- SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion2023
- Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks2022
- Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances2022
- Deep Affordance Foresight: Planning Through What Can Be Done in the Future2021
- Integrated Task and Motion PlanningAnnual Review of Control, Robotics, and Autonomous Systems, 2021
- Learning compositional models of robot skills for task and motion planningThe International Journal of Robotics Research, 2021
- Learning Gradient Fields for Shape Generation2020
- Online Replanning in Belief Space for Partially Observable Task and Motion Problems2020
- PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding2019