Syu-Siang Wang

Speech signals face challenges in the presence of ambient sounds, encompassing factors like reverberation and diffuse noise, leading to compromises in clarity and intelligibility. Taking inspiration from the effectiveness of human binaural hearing, researchers have delved into the exploration of distributed speech enhancement (DSE) processors on the distributed microphone system. Despite the achievements of previous approaches, challenges remain, especially in reducing noise from speech signals and improving model interpretability. This study introduces an innovative geometrically informed graph neural network (GIGNN) designed for DSE tasks. The distinct advantage of GIGNN lies in the capability of GNNs to visualize structured data, proving effective in capturing a wide array of spatiotemporal interactions. Additionally, we assess the effectiveness of geometrically informed spatial matrices within GNNs in our evaluation. Experimental validation in varying signal-to-noise ratios in real-life scenarios underscores the potential of GIGNN.

The rapid evolution of voice technology has heightened the need for robust detection systems to distinguish between authentic and tampered speech. Recent competitions have significantly advanced the development of countermeasures against spoofing attacks. However, while advancements in detection technologies have been notable, existing methods often focus on a single type of tampering and language. Our contribution lies in developing an improved model that integrates an enhanced ResNet architecture with an LSTM to improve the detection of tampered audio, particularly in challenging multilingual scenarios. In the experiments, we built a hybrid dataset from self-recording Chinese speech and public VCTK2 English samples, enhanced the ResNet model generalization capabilities, and evaluated our approach using the bilingual dataset. Experiment results demonstrate that the proposed approach achieves a superior performance with an equal error rate of 11.62%, even in the face of bilingual conditions, and, more importantly, outperforms the leading models from ASVSpoof 2021 and ADD 2022 competitions. We also employed advanced tampering techniques, including CycleGAN voice conversion and auto splicing, to simulate real-world tampering scenarios and verify the effectiveness of the proposed approach.

Voice-related online communication applications are vulnerable to disruptions from complex environments, such as packet loss in IP-switch channels and ambient noise, which hinder communication efficiency. Various packet loss concealment (PLC) and speech enhancement (SE) techniques have been developed to enhance speech quality and to improve user experience. However, generating high-quality speech under noisy conditions with low computational cost remains a significant challenge. This study introduces a lightweight FCN-IPF approach that integrates a fully convolutional network (FCN) with an interpolation-based post-filter (IPF) to reduce signal processing time while improving sound quality and intelligibility. The proposed FCN-IPF is evaluated on system robustness, computational cost, and performance under packet loss and ambient noise conditions. Results show that FCN-IPF-processed speech achieves improvements of 17.65% in sound quality and 12.00% in intelligibility compared to the packet loss input. Under challenging conditions, sound quality and intelligibility are improved by 12.84% and 10.45%, respectively. Additionally, compared to the conventional CRN_FC method, FCN-IPF reduces model processing time per frame from 60.52 ms to 42.69 ms and decreases memory usage from 162 MB to 6.51 MB. These enhancements lower communication latency and boost noise reduction capabilities, making FCN-IPF well-suited for communication in challenging environments.

The utilization of face masks is an essential healthcare measure, particularly during times of pandemics, yet it can present challenges in communication in our daily lives. To address this problem, we propose a novel approach known as the human-in-the-loop StarGAN (HL-StarGAN) face-masked speech enhancement method. HL-StarGAN comprises discriminator, classifier, metric assessment predictor, and generator that leverages an attention mechanism. The metric assessment predictor, referred to as MaskQSS, incorporates human participants in its development and serves as a “human-in-the-loop” module during the learning process of HL-StarGAN. The overall HL- StarGAN model was trained using an unsupervised learning strategy that simultaneously focuses on the reconstruction of the original clean speech and the optimization of human perception. To implement HL-StarGAN, we curated a face-masked speech database named “FMVD,” which comprises recordings from 34 speakers in three distinct face-masked scenarios and a clean condition. We conducted subjective and objective tests on the proposed HL-StarGAN using this database. The outcomes of the test results are as follows: (1) MaskQSS successfully predicted the quality scores of face mask voices, outperforming several existing speech assessment methods. (2) The integration of the MaskQSS predictor enhanced the ability of HL-StarGAN to transform face mask voices into high-quality speech; this enhancement is evident in both objective and subjective tests, outperforming conventional StarGAN and CycleGAN-based systems.

Electroencephalography (EEG) measures changes in neuronal activity and can reveal significant changes from infancy to adulthood concomitant with brain maturation, making it a potential physiological marker of brain maturation and cognition. To investigate a promising deep learning tool for EEG classification, we applied the bidirectional long short-term memory (BLSTM) algorithm to analyze EEG data from the pediatric EEG laboratory of Taipei Tzu Chi Hospital. The trained BLSTM model was 86% accurate when identifying EEGs from young children (8 months–6 years) and adolescents (12–20 years). However, there was only a modest classification accuracy (69.3%) when categorizing EEG samples into three age groups (8 months–6 years, 6–12 years, and 12–20 years). For EEG samples from patients with intellectual disability, the prediction accuracy of the trained BLSTM model was 46.4%, which was significantly lower than its accuracy for EEGs from neurotypical patients, indicating that the individual’s intelligence plays a major role in the age prediction. This study confirmed that scalp EEG can reflect brain maturation and the BLSTM algorithm is a feasible deep learning tool for the identification of cognitive age. The trained model can potentially be applied to clinical services as a supportive measurement of neurodevelopmental status.

The detection of audio tampering plays a crucial role in ensuring the authenticity and integrity of multimedia files. This paper presents a novel approach to identifying tampered audio files by leveraging the unique Electric Network Frequency (ENF) signal, which is inherent to the power grid and serves as a reliable indicator of authenticity. The study begins by establishing a comprehensive Chinese ENF database containing diverse ENF signals extracted from audio files. The proposed methodology involves extracting the ENF signal, applying wavelet decomposition, and utilizing the autoregressive model to train effective classification models. Subsequently, the framework is employed to detect audio tampering and assess the influence of various environmental conditions and recording devices on the ENF signal. Experimental evaluations conducted on our Chinese ENF database demonstrate the efficacy of the proposed method, achieving impressive accuracy rates ranging from 91% to 93%. The results emphasize the significance of ENF-based approaches in enhancing audio file forensics and reaffirm the necessity of adopting reliable tamper detection techniques in multimedia authentication.

Deep learning techniques such as convolutional neural networks (CNN) have been successfully applied to identify pathological voices. However, the major disadvantage of using these advanced models is the lack of interpretability in explaining the predicted outcomes. This drawback further introduces a bottleneck for promoting the classification or detection of voice-disorder systems, especially in this pandemic period. In this paper, we proposed using a series of learnable sinc functions to replace the very first layer of a commonly used CNN to develop an explainable SincNet system for classifying or detecting pathological voices. The applied sinc filters, a front-end signal processor in SincNet, are critical for constructing the meaningful layer and are directly used to extract the acoustic features for following networks to generate high-level voice information. We conducted our tests on three different Far Eastern Memorial Hospital voice datasets. From our evaluations, the proposed approach achieves the highest 7%–accuracy and 9%–sensitivity improvements from conventional methods and thus demonstrates superior performance in predicting input pathological waveforms of the SincNet system. More importantly, we intended to give possible explanations between the system output and the first-layer extracted speech features based on our evaluated results.

This study presents a deep learning-based speech signal-processing mobile application known as CITISEN. The CITISEN can perform three functions: speech enhancement (SE), model adaptation (MA), and background noise conversion (BNC), which allow CITISEN to be used as a platform for utilizing and evaluating SE models and flexibly extend the models to address various noise environments and users. For SE, CITISEN downloads pretrained SE models on the cloud server and then uses these models to effectively reduce noise components from prerecordings or instant recordings provided by users. When it encounters noisy speech signals with unknown speakers or noise types, the MA function allows CITISEN to improve the SE performance effectively. A few audio files of unseen speakers or noise types are recorded and uploaded to the cloud server and then used to adapt the pretrained SE model. Finally, for BNC, CITISEN removes the original background noise using an SE model and then mixes the processed speech signal with new background noise. The novel BNC function can evaluate SE performance under specific conditions, cover people’s tracks, and provide entertainment. The experimental results confirmed the effectiveness of SE, MA, and BNC functions. Compared with the noisy speech signals, the enhanced speech signals by SE achieved about 6% and 33% of improvements, respectively, in terms of short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ). With MA, the STOI and PESQ could be further improved by approximately 6% and 11%, respectively. Note that the SE model and MA method are not limited to the ones described in this study and can be replaced with any SE model and MA method. Finally, the BNC experiment results indicated that the speech signals of original and converted backgrounds have a close scene identification accuracy and similar embeddings in an acoustic scene classification model. Therefore, the proposed BNC can effectively convert the background noise of a speech signal and be a data augmentation method when clean speech signals are unavailable.