Xia Mao

Speech emotion recognition (SER) is an important component of emotion computation and signal processing. Recently, many works have applied abundant acoustic features and complex model architectures to enhance the model’s performance, but these works sacrifice the portability of the model. To address this problem, we propose a model utilizing only the fundamental frequency from electroglottograph (EGG) signals. EGG signals are a sort of physiological signal that can directly reflect the movement of the vocal cord. Under the assumption that different acoustic features share similar representations in the internal emotional state, we propose cross-modal emotion distillation (CMED) to train the EGG-based SER model by transferring robust speech emotion representations from the log-Mel-spectrogram-based model. Utilizing the cross-modal emotion distillation, we achieve an increase of recognition accuracy from 58.98% to 66.80% on the S70 subset of the Chinese Dual-mode Emotional Speech Database (CDESD 7-classes) and 32.29% to 42.71% on the EMO-DB (7-classes) dataset, which shows that our proposed method achieves a comparable result with the human subjective experiment and realizes a trade-off between model complexity and performance.

This paper proposes a framework of applying only the EGG signal for speech synthesis in the limited categories of contents scenario. EGG is a sort of physiological signal which can reflect the trends of the vocal cord movement. Note that EGG’s different acquisition method contrasted with speech signals, we exploit its application in speech synthesis under the following two scenarios. (1) To synthesize speeches under high noise circumstances, where clean speech signals are unavailable. (2) To enable dumb people who retain vocal cord vibration to speak again. Our study consists of two stages, EGG to text and text to speech. The first is a text content recognition model based on Bi-LSTM, which converts each EGG signal sample into the corresponding text with a limited class of contents. This model achieves 91.12% accuracy on the validation set in a 20-class content recognition experiment. Then the second step synthesizes speeches with the corresponding text and the EGG signal. Based on modified Tacotron-2, our model gains the Mel cepstral distortion (MCD) of 5.877 and the mean opinion score (MOS) of 3.87, which is comparable with the state-of-the-art performance and achieves an improvement by 0.42 and a relatively smaller model size than the origin Tacotron-2. Considering to introduce the characteristics of speakers contained in EGG to the final synthesized speech, we put forward a fine-grained fundamental frequency modification method, which adjusts the fundamental frequency according to EGG signals and achieves a lower MCD of 5.781 and a higher MOS of 3.94 than that without modification.

Electroglottograph(EGG) is a physiological signal collected from the throat which reflects the vocal cord movement. EGG signals can be still collected from patients without speaking ability or from the extremely noisy environments. Additionally, the trends of the vocal cord movement will be distinctive for long enough Chinese sentences with different contents. Therefore, it is valuable and possible to carry out the research of applying only the EGG signals for content classification or recognition. In this paper, a content classification method with EGG was proposed, which consists of an EGG feature extraction module and a classification network based on LSTM(Long Short-Term Memory) units. The EGG feature extraction module was composed of three parts: the voiced segments extraction, the feature extraction and the F ₀ smoothing. The classification network was made of a three-layer bidirectional LSTM encoder. This method achieved 91.12% accuracy on the validation set in 20-class content classification experiment, which provides the reference for further study in content classification and recognition with EGG signals.

Human action recognition is an important task. This study presents an efficient framework for recognising action with a 3D skeleton kinematic joint model in less computational time for practical usage. First, a tensor shape descriptor (TSD) is proposed in this study, which takes advantage of the spatial independence of body joints, avoids a lot of difficult problem of the explicit motion estimation required in traditional methods, reserves the spatial information of each frame. Thus, the new TSD is a complete and view‐invariant descriptor. Second, a novel tensor dynamic time warping (TDTW) method is proposed to measure joint‐to‐joint similarity of 3D skeletal body joints locally in the temporal extent, which is implemented by extending DTW to that of two multiway data arrays (or tensors). Then, a multi‐linear projection process is employed to map the TSD to a low‐dimensional tensor subspace, which is classified by the nearest neighbour classifier. The experiment results on the public action data set (MSR‐Action3D) and motion capture data set (CMU_Mocap) show that the proposed method can achieve a comparable or better performance in recognition accuracy compared with the state‐of‐the‐art approaches.

Traditional studies in vision‐based hand gesture recognition remain rooted in view‐dependent representations, and hence users are forced to be fronto‐parallel to the camera. To solve this problem, view‐invariant gesture recognition aims to make the recognition result independent of viewpoint changes. However, in current works the view‐invariance is achieved at the price of mixing different gesture patterns that have similar trajectory curve shape but different semantic meanings. For example, the gesture ‘push’ can be mistaken as ‘drag’ from another viewpoint. To address this shortcoming, in this study, the authors use a shape descriptor to extract the view‐invariant features of a three‐dimensional (3D) trajectory. As the shape features are invariant to omnidirectional viewpoint changes, the orientation features are then added into weight different rotation angles so that similar trajectory shapes are better separated. The proposed method was conducted on two different databases, including a popular Australian Sign Language database and a challenging Kinect Hand Trajectory database. Experimental results show that the proposed algorithm achieves a higher average recognition rate than the state‐of‐the‐art approaches, and can better distinguish confusing gestures while meeting the view‐invariant condition.

Motion trajectory recognition is important for characterizing the moving property of an object. The speed and accuracy of trajectory recognition rely on a compact and discriminative feature representation, and the situations of varying rotation, scaling and translation has to be specially considered. In this paper we propose a novel feature extraction method for trajectories. Firstly a trajectory is represented by a proposed point context, which is a rotation-scale-translation (RST) invariant shape descriptor with a flexible tradeoff between computational complexity and discrimination, yet we prove that it is a complete shape descriptor. Secondly, the shape context is nonlinearly mapped to a subspace by kernel nonparametric discriminant analysis (KNDA) to get a compact feature representation, and thus a trajectory is projected to a single point in a low-dimensional feature space. Experimental results show that, the proposed trajectory feature shows encouraging improvement than state-of-art methods.