Ning Jia

Aiming at the problem of blurred pronunciation caused by dysarthria in stroke patients, we propose a non parallel corpus speech articulation method based on Fre-MaskCycleGAN-VC. The method consists of three core stages: 1) Aiming at the coexistence of fuzzy speech segments and clear retention segments in dysarthria speech of stroke patients, dynamic speech segmentation preprocessing based on equivalent sound level (Leq) is used to accurately locate the fuzzy segments that need to be enhanced; 2) Feature extraction combining dynamic mask and retro production statistical features; 3) The resolution connected generator and resolution wise discriminators architecture that integrate the frequency processing model. Multiple groups of experiments were carried out on the stroke dysarthria speech data set. The experimental results show that the Fre-MaskCycleGAN-VC method has significantly improved the naturalness of speech (Mean Opinion Score (MOS) increased by 14.2%), intelligibility (WA increased by 2.6%) and timbre fidelity (MFCC correlation coefficient 0.92, F0 error rate 4.2%). Phased evolution experiments show that the model can generate four gradual repair versions from heavily blurred speech to near healthy speech, and the repair effect of grade 2–3 is better than that of the original healthy speech. Through multi-stage feature processing and adversarial training mechanism, we provide a clear speech generation scheme that retains the original timbre for patients with dysarthria

As one of the most challenging and promising topics in speech field, emotion speech synthesis is a hot topic in current research. At present, the emotion expression ability, synthesis speed and robustness of synthetic speech need to be improved. Cycle-consistent Adversarial Networks (CycleGAN) provides a two-way breakthrough in the transformation of emotional corpus information. But there is still a gap between the real target and the synthesis speech. In order to narrow this gap, we propose an emotion speech synthesis method combining multi-channel Time–frequency Domain Generative Adversarial Networks (MC-TFD GANs) and Mixup. It includes three stages: multichannel Time–frequency Domain GANs (MC-TFD GANs), loss estimation based on Mixup and effective emotion region stacking based on Mixup. Among them, the gating unit GTLU (gated tanh linear units) and the image expression method of speech saliency region are designed. It combines the Time–frequency Domain MaskCycleGAN based on improved GTLU and the time-domain CycleGAN based on saliency region to form the multi-channel GAN in the first stage. Based on Mixup method, the calculation method of loss and the aggravation degree of emotion region are designed. Compared with several popular speech synthesis methods, the comparative experiments were carried out on the interactive emotional dynamic motion capture (IEMOCAP) corpus. The bi-directional three-layer long short-term memory (LSTM) model was used as the verification model. The experimental results showed that the mean opinion score (MOS) and the unweighted accuracy (UA) of the speech generated by the synthesis method were improved, and the improvements were 4% and 2.7%, respectively. The current model was superior to the existing GANs model in subjective evaluation and objective experiments, ensure that the speech generated by this model had higher reliability, better fluency and emotional expression ability.

The generation of emotional speech is a challenging and widely applied research topic in the field of speech processing. Because the design method of effective speech feature expression and generation model directly affects the accuracy of emotional speech generation, it is difficult to find a general solution of emotional speech synthesis. In this paper, the CycleGAN model is used as the starting point, and the improved convolution neural network (CNN) model and identity mapping loss scheme are used to achieve effective timing information capture. At the same time, we learn the positive mapping and the reverse mapping to find the best matching design scheme, and retain the speech information in this process, without relying on other audio data. Experiments show that the emotional speech can be accurately recognized by comparing the speech emotion before and after the improvement on the speech corpus of children’s reading. By comparing with the common emotional speech generation model, the advantages of the model proposed in this paper are verified.