Won Dong Kim

Humans sense and interpret touches on their skin for social interaction. Similarly, robotic systems with human-like robotic skin can intuitively interact with humans. Therefore, many tactile sensors have been developed, but their excessive sensing elements, narrow sensing bandwidth, and fragility limit their applications as robotic skin. This article proposes a robotic skin structure that mimics the human skin layer and Pacinian corpuscle using a textured resilient fabric, a uniquely structured airmesh, and encapsulated microphones. Furthermore, functions such as tactile stimulus encoding, tactile stimulus dispersion, elastomechanical properties, and wide sensitivity bandwidth are mimicked. The developed skin identifies tactile locations and patterns using a small number of sensing nodes and algorithms that interpret tactile sensations. These include passive acoustic tomography to localize touch, signal intensities map, and spectrogram to encode spatiotemporal characteristics of touch, and convolutional neural network to decode and classify touch. As a result, the algorithms localized tactile stimulus with a mean error of 1.8 cm and classified touch into nine classes with an accuracy of 93.3%. Furthermore, the developed robotic skin has no rigid material and employs a few sensing nodes, thus easily accommodating large nonplanar surfaces. The skin was implemented on a robotic arm to demonstrate a physical human–robot interaction and on a vertically cylindrical surface of similar size to a social robot to demonstrate the scaling up to larger systems with a lower sensing node density.

Dexterous robotic grasping gains great benefits from tactile sensation, but delicate object exploration by tactile information is challenged by difficulty in rich and efficient data production. In this letter, we propose a tactile-based in-hand object perception approach, empowered by a sim-to-real approach toward a data-efficient learning process. A high-fidelity tactile input, measured by a pair of vision-based tactile sensors, was represented as a point cloud facilitating dual functionality of object classification and the associated pose estimation. For the classification, we constructed PoinTacNet , a variation of PointNet to fit into tactile data processing. A reliable simulation methodology on tactile input was employed for the pretraining of the model, transferred to the fine-tuning process facing real tactile data. Taking inspiration from human behaviors, a re-grasping strategy was imparted by means of conditional accumulation of class likelihood distribution. The result of the framework facilitates a high object classification accuracy of 83.89 $\%$ on the ten objects from McMaster-Carr's CAD models, which is significantly improved by the re-grasping. In addition, a set of benchmarks displays the computational efficiency in the sim-to-real transfer. In line with the successful classification, the posture of in-hand objects is estimated using point cloud registration algorithms, achieving an average angular and translational RMSE of 5.03 $^\circ$ and 2.41 mm, respectively. The proposed approach has the potential to enable robots to attain human-like haptic exploration skills for perceiving unstructured environments.

Data-driven methods have been successfully applied to images from vision-based tactile sensors to fulfill various manipulation tasks. Nevertheless, these methods remain inefficient because of the lack of methods for simulating the sensors. Relevant research on simulating vision-based tactile sensors generally focus on generating images without markers, owing to the challenges in accurately generating marker motions caused by elastomer deformation. This disallows access to tactile information deducible from markers. In this letter, we propose a generative adversarial network (GAN)-based method to generate realistic marker-embedded tactile images in GelSight-like vision-based tactile sensors. We trained the proposed GAN model with an aligned real tactile and simulated depth image dataset obtained from deforming the sensor against various objects. This allows the model to translate simulated depth image sequences into RGB tactile images with markers. Furthermore, the generator in the proposed GAN allows the network to integrate the history of deformations from the depth image sequences to generate realistic marker motions during the normal and lateral sensor deformations. We evaluated and compared the positional accuracy of the markers and image similarity metrics of the images generated via our method with those from prior methods. The generated tactile images from the proposed model show a 28.3% decrease in marker positional error and a 93.5% decrease in the image similarity metric (MSE) compared to those generated by previous methods, validating the effectiveness of our approach.

Vision-based tactile sensors provide diverse information of external tactile stimuli on the skin of sensors using both marker and reflective membrane images. However, when markers and reflective membranes are used concurrently, conventional opaque markers inevitably disturb the camera’s view of the reflective membrane. Thus, simultaneously increasing the quality of tactile information extracted from each visual feature has remained a challenge. In this study, we present a tactile sensing finger, UVtac, that utilizes switchable ultraviolet (UV) markers to decouple the marker and reflective membrane images to offer three-axis force estimation and object localization, whose performances are unaffected by each other. Our UVtac showed improved force estimation performance by using larger-sized UV markers through quantitative evaluation. The UVtac with 1.2 mm diameter markers showed a root mean square error of 0.264 N in estimating normal forces up to 10 N and 0.219 N in estimating the shear forces up to 5 N, when indented with an 8 × 8 mm $^{2}$ square tooltip. Based on the object localization experiment, the UVtac was verified to have a 31 % lower root mean square error than the case using opaque black markers. Finally, we demonstrated object alignment and contact force-tracking tasks using the UVtac to emphasize its multifunctionality.