Soft Contact Simulation and Manipulation Learning of Deformable Objects With Vision-Based Tactile Sensor
- 23 June 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 17618-17630
Abstract
Deformable object manipulation is a challenging problem due to its complex deformable properties. With the development of artificial intelligence, learning-based methods have shown outstanding performance in robotic manipulation. Previous works have investigated the manipulation of deformable objects via Reinforcement Learning (RL) in simulation. However, they approximate object deformation with particles, using particle states as observations, which are unavailable in reality. To address these issues, we utilize Vision-Based Tactile Sensors (VBTSs) as the end-effector to manipulate and observe the deformable objects. In this work, we develop a new contact simulation environment for deformable objects, including elastic, plastic, and elastoplastic. We utilize RL strategies and expert demonstrations to train agents in the simulation. Finally, we build a real experimental platform to complete the sim-to-real tasks and robustness testing. Our work introduces an innovative strategy that utilizes high-resolution VBTSs for contact simulation and manipulation of deformable objects. The experimental results show superior performances of deformable object manipulation with the proposed method.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62173197)
- Beijing Natural Science Foundation (L222012)
- National High Level Hospital Clinical Research Funding (2022-PUMCH-B-002)
- Zhipu Company
This publication has 40 references indexed in Scilit:
- Tactile data generation and applications based on visuo-tactile sensors: A reviewInformation Fusion, 2025
- DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects2024
- A compact fingertip-shaped visuo-tactile sensor for manipulator adaptive grasping controlSensors and Actuators A: Physical, 2024
- SofaGym: An Open Platform for Reinforcement Learning Based on Soft Robot SimulationsSoft Robotics, 2023
- Non-destructive Fruit Firmness Evaluation Using Vision-Based Tactile Information2022
- Learning Shape Control of Elastoplastic Deformable Linear Objects2021
- Introduction to Reinforcement Learning2020
- ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition2018
- The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic MorphologiesSoft Robotics, 2018
- Four Methods for Roundness EvaluationPhysics Procedia, 2012