PoCoDP3: Pose- and Contact-Aware Visual-Tactile Policy for Contact-Rich 3D Manipulation
- 12 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (3) , 2434-2441
Abstract
Imitation learning in contact-rich tasks requires both global spatial awareness and fine-grained in-hand interaction understanding. However, vision-only policies based on images or point clouds are often susceptible to occlusion and struggle to capture critical contact details, particularly in visually ambiguous regions or during subtle tactile interactions. In this work, we present PoCoDP3, a pose- and contact-aware visual-tactile policy that integrates 3D point clouds and tactile inputs to generate actions in contact-rich tasks. PoCoDP3 introduces a dual-branch tactile encoder that jointly models contact dynamics and estimates in-hand object pose, enabling structured tactile representations for precise contact-rich manipulation. A contact-driven cross-modal fusion mechanism adaptively prioritizes sensory modalities based on real-time interaction cues, enabling efficient visual-tactile integration. Moreover, a reference-guided diffusion policy leverages reference action offsets to reduce sampling steps, significantly accelerating inference while maintaining action quality. Experiments across simulation and real-world tasks demonstrate that PoCoDP3 consistently outperforms representative 2D and 3D policies in terms of both accuracy and inference efficiency.Keywords
Funding Information
- Basic and Applied Basic Research Foundation of Guangdong Province (2025A1515010397)
- Project of the Joint Research and Development Center for Future Robotics and Artificial Intelligence
- Southeast University
This publication has 20 references indexed in Scilit:
- Visual–tactile learning of robotic cable-in-duct installation skillsAutomation in Construction, 2024
- TARS: Tactile Affordance in Robot Synesthesia for Dexterous ManipulationIEEE Robotics and Automation Letters, 2024
- The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning2024
- RISE: 3D Perception Makes Real-World Robot Imitation Simple and Effective2024
- Residual Denoising Diffusion Models2024
- FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-Motor Robot Manipulation SkillsIEEE Robotics and Automation Letters, 2024
- Active Extrinsic Contact Sensing: Application to General Peg-in-Hole Insertion2022
- VisuoTactile 6D Pose Estimation of an In-Hand Object Using Vision and Tactile Sensor DataIEEE Robotics and Automation Letters, 2022
- Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements2021
- An Approach for Imitation Learning on Riemannian ManifoldsIEEE Robotics and Automation Letters, 2017