Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion
- 9 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (4) , 4417-4424
Abstract
Acting in cluttered environments requires predicting and avoiding collisions while still achieving precise control. Conventional optimization-based controllers can enforce physical constraints, but they struggle to produce feasible solutions quickly when many obstacles are present. Diffusion models can generate diverse trajectories around obstacles, yet prior approaches lacked a general and efficient way to condition them on scene structure. In this paper, we show that combining diffusion-based warm-starting conditioned with a latent object-centric representation of the scene and with a collision-aware model predictive controller (MPC) yields reliable and efficient motion generation under strict time limits. Our approach conditions a diffusion transformer on the system state, task, and surroundings, using an object-centric slot attention mechanism to provide a compact obstacle representation suitable for control. The sampled trajectories are refined by an optimal control problem that enforces rigid-body dynamics and signed-distance collision constraints, producing feasible motions in real time. On benchmark tasks, this hybrid method achieved markedly higher success rates and lower latency than sampling-based planners or either component alone. Real-robot experiments with a torque-controlled Panda confirm reliable and safe execution with MPC. An open-source implementation is provided here.Keywords
All Related Versions
Funding Information
- European project AGIMUS (101070165)
- Artificial and Natural Intelligence Toulouse Institute (ANR-19-P3IA-0004)
- ELLIOT (101214398)
This publication has 35 references indexed in Scilit:
- Diffusion-based Generation, Optimization, and Planning in 3D Scenes2023
- Crocoddyl: An Efficient and Versatile Framework for Multi-Contact Optimal Control2020
- Learning How to Walk: Warm-starting Optimal Control Solver with Memory of Motion2020
- Motion Planning Networks2019
- Using a Memory of Motion to Efficiently Warm-Start a Nonlinear Predictive Controller2018
- Deep Residual Learning for Image Recognition2016
- Model predictive control: Recent developments and future promiseAutomatica, 2014
- STOMP: Stochastic trajectory optimization for motion planning2011
- AprilTag: A robust and flexible visual fiducial system2011
- Differential Dynamic Programming–A Unified Approach to the Optimization of Dynamic Systems* *This work was done during the author's visit to the Division of Engineering and Applied Physics, Harvard University, and was supported by the U.S. Army Research Office, the U.S. Air Force Office of Scientific Rearch and the U.S. Office of Naval Research under the Joint Services Electronics Program by Contracts N00014-67-A-0298-0006, 0005. and 0008.1973