Yun-Kai Li

Multimodal emotion recognition (MER) plays a vital role in human-computer interaction for consumer electronics. However, a key challenge to the practical deployment of current MER methods is the presence of varying noise levels and potential conflicts among modalities. Moreover, existing multimodal learning approaches usually evaluate modality quality from a global perspective, overlooking the inherent data quality biases present across different modalities. To this end, a multimodal adaptive reweighting strategy (Mars) is proposed, which combines an uncertainty weighting mechanism with a staged training framework for reliable multimodal fusion. Specifically, the proposed uncertainty weighting mechanism computes a dynamically scaled loss for each sample pair, enabling the model to automatically prioritize information from more reliable modalities while de-emphasizing less reliable ones from a sample perspective. The staged training framework initially emphasizes learning from the global sample distribution and gradually shifts focus to high-quality samples in later stages, thereby mitigating overfitting. This approach enhances the robustness of each modality and improves overall MER performance. Experimental results show that Mars outperforms baseline methods by 4.6% and 6.07% on the RAVDESS and FETE datasets, respectively, without introducing additional model parameters. The code of Mars is available at: https://github.com/AI-touch/Mars.

During human-robot interaction, touch gesture and emotion recognition (TGER) can help robots perceive and infer human intentions and emotional states, and establish a more harmonious human-robot relationship. Conventional handcrafted feature-based machine learning methods and end-to-end deep learning networks have been used for TGER. However, existing TGER methods are mainly based on single-task models, i.e., the touch gestures and emotions are recognized separately with different models, which suffer from relatively low recognition accuracy, high computational complexity, or high memory burden. To solve these existing problems, this paper proposed a multi-scale spatiotemporal convolutions with attention mechanism for touch gesture and emotion recognition (MUSCAT), and established a multi-task TGER framework for an overall model construction. MUSCAT uses a factorized spatiotemporal convolutional neural network to sequentially extract spatial and temporal features, and multi-scale convolution is used in temporal convolution. The attention mechanisms are applied after spatial convolution, temporal convolution and point-wise convolution to respectively focus on the key information of spatial, temporal and channel dimensions. The proposed MUSCAT model has a deeper structure than the standard three-dimensional convolution, which gains better performance with fewer model parameters. Extensive experiments were conducted on two touch datasets, and the results verify the feasibility of the proposed method.

Touch gesture recognition (TGR) plays a pivotal role in many applications, such as socially assistive robots and embodied telecommunication. However, one obstacle to practicality of existing TGR methods is the individual disparities across subjects. Moreover, a deep neural network trained with multiple existing subjects can easily lead to over fitting for a new subject. Hence, how to mitigate the discrepancies between the new and existing subjects and establish a generalized network for TGR is a significant task to realize reliable human-robot tactile interaction. In this paper, a novel framework for Multi-source domain Adaption via Shared-Specific feature projection (MASS) is proposed, which incorporates intra-domain discriminant, multi-domain discriminant, and cross-domain consistency into a deep learning network for cross-subject TGR. Specifically, the MASS method first extracts the shared features in the common feature space of training subjects, with which a domain-general classifier is built. Then, the specific features of each pair of training and testing subjects are mapped and aligned in their common feature space, and multiple domain-specific classifiers are trained with the specific features. Finally, the domain-general classifier and domain-specific classifiers are ensembled to predict the label for the touch samples of a new subject. Experimental results performed on two datasets show that our proposed MASS method achieves remarkable results for cross-subject TGR. The code of MASS is available at: https://github.com/AI-touch/MASS.

Touch is one of the most essential and effective means to convey affective feelings and intentions in human communication. For a social robot, the ability to recognize human touch gestures and emotions could help realize efficient and natural human–robot interaction. To this end, an affective touch gesture dataset involving ten kinds of touch gestures and 12 kinds of discrete emotions was built by using a pressure sensor array, in which the acquired touch gesture samples are three-dimensional (3-D) spatiotemporal signals that include the shape appearance and motion dynamics. Due to the excellent performance of convolutional neural networks (CNNs), spatiotemporal CNNs have been effectively verified by researchers for 3-D signal classification. However, the large number of parameters and the high complexity of training 3-D convolution kernels remain to be solved. In this article, a decomposed spatiotemporal convolution was designed for feature representation from the raw touch gesture samples. Specifically, the 3-D kernel was factorized into three 1-D kernels by tensor decomposition. The proposed convolution has a simpler but deeper architecture than standard 3-D convolution, which improves the nonlinear expression ability of the model. Besides, the computation cost can be reduced without compromising recognition accuracy. Using a user-dependent test mode, the proposed method yields the accuracies of up to 92.41% and 72.47% for touch gesture and emotion recognitions, respectively. Experimental results demonstrate the effectiveness of the proposed method, and at the same time, preliminarily verify the feasibility of robot perceiving human emotions through touch.

The touch gesture is one of the most essential and effective means to transfer affective feelings and intents in humans’ communication. For an intelligent agent or a robot, the ability to automatically detect and recognize human touch can realize efficient and natural human–robot interaction. To this end, a novel spatiotemporal fusion feature extraction method is proposed for touch gesture classification tasks. The proposed method extracts time-frequency features from wavelet coefficients based on discrete wavelet transforms. Then, the feature array of space and frequency bands is constructed to extract the spatiotemporal fusion features. A publicly available touch gesture dataset called CoST is used to perform the touch gesture recognition. The recognition result of 14 gesture classes using a user-independent model yields an accuracy of up to 64.17%. Experimental results show that this method outperforms the state-of-the-art ones and that the spatiotemporal fusion features effectively boost the performance of touch gesture recognition.