Tian-Hao Yang

During human-robot interaction, touch gesture and emotion recognition (TGER) can help robots perceive and infer human intentions and emotional states, and establish a more harmonious human-robot relationship. Conventional handcrafted feature-based machine learning methods and end-to-end deep learning networks have been used for TGER. However, existing TGER methods are mainly based on single-task models, i.e., the touch gestures and emotions are recognized separately with different models, which suffer from relatively low recognition accuracy, high computational complexity, or high memory burden. To solve these existing problems, this paper proposed a multi-scale spatiotemporal convolutions with attention mechanism for touch gesture and emotion recognition (MUSCAT), and established a multi-task TGER framework for an overall model construction. MUSCAT uses a factorized spatiotemporal convolutional neural network to sequentially extract spatial and temporal features, and multi-scale convolution is used in temporal convolution. The attention mechanisms are applied after spatial convolution, temporal convolution and point-wise convolution to respectively focus on the key information of spatial, temporal and channel dimensions. The proposed MUSCAT model has a deeper structure than the standard three-dimensional convolution, which gains better performance with fewer model parameters. Extensive experiments were conducted on two touch datasets, and the results verify the feasibility of the proposed method.

Touch gesture recognition (TGR) plays a pivotal role in many applications, such as socially assistive robots and embodied telecommunication. However, one obstacle to practicality of existing TGR methods is the individual disparities across subjects. Moreover, a deep neural network trained with multiple existing subjects can easily lead to over fitting for a new subject. Hence, how to mitigate the discrepancies between the new and existing subjects and establish a generalized network for TGR is a significant task to realize reliable human-robot tactile interaction. In this paper, a novel framework for Multi-source domain Adaption via Shared-Specific feature projection (MASS) is proposed, which incorporates intra-domain discriminant, multi-domain discriminant, and cross-domain consistency into a deep learning network for cross-subject TGR. Specifically, the MASS method first extracts the shared features in the common feature space of training subjects, with which a domain-general classifier is built. Then, the specific features of each pair of training and testing subjects are mapped and aligned in their common feature space, and multiple domain-specific classifiers are trained with the specific features. Finally, the domain-general classifier and domain-specific classifiers are ensembled to predict the label for the touch samples of a new subject. Experimental results performed on two datasets show that our proposed MASS method achieves remarkable results for cross-subject TGR. The code of MASS is available at: https://github.com/AI-touch/MASS.

Touch is one of the most essential and effective means to convey affective feelings and intentions in human communication. For a social robot, the ability to recognize human touch gestures and emotions could help realize efficient and natural human–robot interaction. To this end, an affective touch gesture dataset involving ten kinds of touch gestures and 12 kinds of discrete emotions was built by using a pressure sensor array, in which the acquired touch gesture samples are three-dimensional (3-D) spatiotemporal signals that include the shape appearance and motion dynamics. Due to the excellent performance of convolutional neural networks (CNNs), spatiotemporal CNNs have been effectively verified by researchers for 3-D signal classification. However, the large number of parameters and the high complexity of training 3-D convolution kernels remain to be solved. In this article, a decomposed spatiotemporal convolution was designed for feature representation from the raw touch gesture samples. Specifically, the 3-D kernel was factorized into three 1-D kernels by tensor decomposition. The proposed convolution has a simpler but deeper architecture than standard 3-D convolution, which improves the nonlinear expression ability of the model. Besides, the computation cost can be reduced without compromising recognition accuracy. Using a user-dependent test mode, the proposed method yields the accuracies of up to 92.41% and 72.47% for touch gesture and emotion recognitions, respectively. Experimental results demonstrate the effectiveness of the proposed method, and at the same time, preliminarily verify the feasibility of robot perceiving human emotions through touch.