Michael A. Lin

When humans socially interact with another agent (e.g., human, pet, or robot) through touch, they do so by applying varying amounts of force with different directions, locations, contact areas, and durations. While previous work on touch gesture recognition has focused on the spatio-temporal distribution of normal forces, we hypothesize that the addition of shear forces will permit more reliable classification. We present a soft, flexible skin with an array of tri-axial tactile sensors for the arm of a person or robot. We use it to collect data on 13 touch gesture classes through user studies and train a Convolutional Neural Network (CNN) to learn spatio-temporal features from the recorded data. The network achieved a recognition accuracy of 74% with normal and shear data, compared to 66% using only normal force data. Adding distributed shear data improved classification accuracy for 11 out of 13 touch gesture classes.

We present a highly conformable stretchable sensory skin made of soft components. The skin uses low pressure pneumatic taxels and stretchable channels to conduct pressure signals to off-board micro-electromechanical systems (MEMS) pressure sensors. The force sensing range captures down to 0.01 N and has a roll-off frequency of 102 Hz. Due to pressure equilibration, the signals are substantially unaffected as the channels stretch and flex over robot joints with up to 100% elongation. The design permits additional floating taxels which, though less insensitive to stretch, provide sensing of contacts across joints and gaps. With no conductive elements or rigid components, the skin is robust to impacts and withstands many cycles of stretch and immersion in water. We apply the skin to a robotic wrist with two intersecting axes and show that it has sufficient sensitivity and bandwidth to detect the onset of sliding as the robot contacts objects. We demonstrate the skin in object acquisition tasks in a constrained environment for which extraneous contacts are unavoidable.

We present a new gripper and exploration approach that uses a finger with very low reflected inertia for probing and then grasping objects. The finger employs a transparent transmission, resulting in a light touch when contact occurs. The finger elements are stiff and mounted on precise Cartesian axes for accurate proprioceptive sensing. Experiments show that the finger can safely move faster into contacts than industrial parallel jaw grippers or even most force-controlled grippers with backdrivable transmissions. This property allows rapid proprioceptive probing of objects. Contact information is leveraged to execute grasping actions with a contact-first strategy and to reduce environment state uncertainty. We evaluate a particle filtering algorithm that inputs contact information from either proprioception, or a combination of tactile sensing and proprioception, to estimate object location. Both methods can estimate location within 2 mm; combined tactile sensing and proprioception requires fewer observations.

Augmented reality (AR) systems have been proposed for medical needle procedures as alternatives to restore the physician's natural perspective and head motion parallax that are lost when using standard imaging methods. AR systems have assumed a rigid needle, but in practice, a needle can experience large deflections that can lead to significant errors. We combine an instrumented needle that measures bending deflections with optical tracking of the needle base for precision and an AR system to provide users with precise three-dimensional rendering of needle motion and deflection inside a body. In controlled tests, users guided a needle through 7 cm of tissue phantom to reach a 2-mm diameter target. We found that displaying needle shape allows users significantly reduce placement errors when compared to the rigid needle assumption. In addition, a new display technique called tip tangent extrapolation, based on the sensed needle endpoint and direction, allowed users to further reduce errors resulting in 26% less error when compared to a rigid needle assumption.