Tac2Pose: Tactile object pose estimation from the first touch
Open Access
- 11 September 2023
- journal article
- research article
- Published by SAGE Publications in The International Journal of Robotics Research
- Vol. 42 (13) , 1185-1209
Abstract
In this paper, we present Tac2Pose, an object-specific approach to tactile pose estimation from the first touch for known objects. Given the object geometry, we learn a tailored perception model in simulation that estimates a probability distribution over possible object poses given a tactile observation. To do so, we simulate the contact shapes that a dense set of object poses would produce on the sensor. Then, given a new contact shape obtained from the sensor, we match it against the pre-computed set using an object-specific embedding learned using contrastive learning. We obtain contact shapes from the sensor with an object-agnostic calibration step that maps RGB (red, green, blue) tactile observations to binary contact shapes. This mapping, which can be reused across object and sensor instances, is the only step trained with real sensor data. This results in a perception model that localizes objects from the first real tactile observation. Importantly, it produces pose distributions and can incorporate additional pose constraints coming from other perception systems, multiple contacts, or priors. We provide quantitative results for 20 objects. Tac2Pose provides high accuracy pose estimations from distinctive tactile observations while regressing meaningful pose distributions to account for those contact shapes that could result from different object poses. We extend and test Tac2Pose in multi-contact scenarios where two tactile sensors are simultaneously in contact with the object, as during a grasp with a parallel jaw gripper. We further show that when the output pose distribution is filtered with a prior on the object pose, Tac2Pose is often able to improve significantly on the prior. This suggests synergistic use of Tac2Pose with additional sensing modalities (e.g., vision) even in cases where the tactile observation from a grasp is not sufficiently discriminative. Given a coarse estimate of an object’s pose, even ambiguous contacts can be used to determine an object’s pose precisely. We also test Tac2Pose on object models reconstructed from a 3D scanner, to evaluate the robustness to uncertainty in the object model. We show that even in the presence of model uncertainty, Tac2Pose is able to achieve fine accuracy comparable to when the object model is the manufacturer’s CAD (computer-aided design) model. Finally, we demonstrate the advantages of Tac2Pose compared with three baseline methods for tactile pose estimation: directly regressing the object pose with a neural network, matching an observed contact to a set of possible contacts using a standard classification neural network, and direct pixel comparison of an observed contact with a set of possible contacts. Website: mcube.mit.edu/research/tac2pose.htmlKeywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Touch based localization of parts for high precision manufacturing2017
- Deep Residual Learning for Image Recognition2016
- In-Hand Object Pose Estimation Using Covariance-Based Tactile To Geometry MatchingIEEE Robotics and Automation Letters, 2016
- Pose estimation for planar contact manipulation with manifold particle filtersThe International Journal of Robotics Research, 2015
- Global estimation of an object’s pose using tactile sensingAdvanced Robotics, 2015
- A new method of constructing a grid in the space of 3D rotations and its applications to texture analysisModelling and Simulation in Materials Science and Engineering, 2014
- Online in-hand object localization2013
- Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensingThe International Journal of Robotics Research, 2013
- Efficient touch based localization through submodularity2013
- Generating Uniform Incremental Grids on SO(3) Using the Hopf FibrationThe International Journal of Robotics Research, 2009