Learning Non-Prehensile Manipulation With Force and Vision Feedback Using Optimization-Based Demonstrations
- 19 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (3) , 2905-2912
Abstract
Non-prehensile manipulation is challenging due to complex contact interactions between objects, the environment, and robots. Model-based approaches can efficiently generate complex trajectories of robots and objects under contact constraints. However, they tend to be sensitive to model inaccuracies and require access to privileged information (e.g., object mass, size, pose), making them less suitable for novel objects. In contrast, learning-based approaches are typically more robust to modeling errors but require large amounts of data. In this letter, we bridge these two approaches to propose a framework for learning closed-loop non-prehensile manipulation. By leveraging computationally efficient Contact-Implicit Trajectory Optimization (CITO), we design demonstration-guided deep Reinforcement Learning (RL), leading to sample-efficient learning. We also present a sim-to-real transfer approach using a privileged training strategy, enabling the robot to perform non-prehensile manipulation using only proprioception, vision, and force sensing without access to privileged information. Our method is evaluated on several tasks, demonstrating that it can successfully perform zero-shot sim-to-real transfer.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- From Imitation to Refinement - Residual Rl for Precise Assembly2025
- Is Linear Feedback on Smoothed Dynamics Sufficient for Stabilizing Contact-Rich Plans?2025
- Hierarchical Contact-Rich Trajectory Optimization for Multi-Modal Manipulation Using Tight Convex Relaxations2025
- Adaptive Compliance Policy: Learning Approximate Compliance for Diffusion Guided Control2025
- Tactile Tool Manipulation2023
- Learning Generalizable Pivoting Skills2023
- Non-prehensile Planar Manipulation via Trajectory Optimization with Complementarity Constraints2022
- Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation2018
- Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates2017
- MuJoCo: A physics engine for model-based control2012