Seung-Eun Kim

Despite their success in speech processing, neural networks often operate as black boxes, prompting the following questions: What informs their decisions, and how can we interpret them? This work examines this issue in the context of lexical stress. A dataset of English disyllabic words was automatically constructed from read and spontaneous speech. Several convolutional neural network (CNN) architectures were trained to predict stress position from a spectrographic representation of disyllabic words lacking minimal stress pairs (e.g., initial stress WAllet, final stress exTEND), achieving up to 92% accuracy on held-out test data. Layerwise relevance propagation, a technique for neural network interpretability analysis, revealed that predictions for held-out minimal pairs (PROtest vs proTEST) were most strongly influenced by information in stressed versus unstressed syllables, particularly the spectral properties of stressed vowels. However, the classifiers also attended to information throughout the word. A feature-specific relevance analysis is proposed, and its results suggest that the best-performing classifier is strongly influenced by the stressed vowel's first and second formants, with some evidence that its pitch and third formant also contribute. These results reveal deep learning's ability to acquire distributed cues to stress from naturally occurring data, extending traditional phonetic work based around highly controlled stimuli.

When listeners encounter a difficult-to-understand talker in a difficult-to-understand situation, their perceptual mechanisms can adapt, making the talker in the situation easier to understand. This study examined talker-specific perceptual adaptation experimentally by embedding speech from second-language (L2) English talkers in varying levels of noise and collecting transcriptions from first-language English listeners (ten talkers, 100 listeners per experiment). Experiments 1 and 2 demonstrated that prior experience with a L2 talker's speech presented first without noise and then with gradually increasing levels of noise facilitated recognition of that talker in loud noise. Experiment 3 tested whether adaptation is driven by tuning-in to the talker's voice and speech patterns, by examining recognition of speech-in-loud-noise following experience with the talker in quiet. Finally, experiment 4 tested whether adaptation is driven by tuning-out the background noise, by measuring speech-in-loud-noise recognition after experience with the talker in consistently loud noise. The results showed that both tuning-in to the talker and tuning-out the noise contribute to talker-specific perceptual adaptation to L2 speech-in-noise.

Researchers have generally assumed that listeners perceive speech compositionally, based on the combined processing of local acoustic–phonetic cues a

Listeners can use lexical information to drive adaptation to talker-specific speech characteristics with a small amount of exposure. This suggests that adaptation may be strongest when listeners are initially exposed to speech in a condition that facilitates word recognition. This was tested by examining the intelligibility of second-language (L2) speech-in-noise when stimuli were initially presented in Quiet versus low signal-to-noise ratio (SNR) conditions. Recordings of 10 L2 English talkers were tested. For each talker, 120 sentences were mixed with speech-shaped noise ranging from −4 dB to 8 dB in steps of 2 dB and in Quiet (15 sentences at each SNR). These stimuli were presented from lowest to highest SNR (forward; −4 dB to Quiet) in one group of L1 English listeners (n = 10/talker) and from highest to lowest SNR (reverse; Quiet to −4 dB) in the other (n = 10/talker). The results showed that early exposure to a clean signal led to better speech perception. At lower SNRs, word recognition accuracy in the reverse order was significantly higher than the forward order, resulting in a steeper slope of psychometric function of intelligibility in the forward versus reverse condition. The quality of initial speech input may influence talker adaptation, particularly in adverse listening conditions.

Measuring how well human listeners recognize speech under varying environmental conditions (speech intelligibility) is a challenge for theoretical, technological, and clinical approaches to speech communication. The current gold standard—human transcription—is time- and resource-intensive. Recent advances in automatic speech recognition (ASR) systems raise the possibility of automating intelligibility measurement. This study tested 4 state-of-the-art ASR systems with second language speech-in-noise and found that one, whisper, performed at or above human listener accuracy. However, the content of whisper's responses diverged substantially from human responses, especially at lower signal-to-noise ratios, suggesting both opportunities and limitations for ASR--based speech intelligibility modeling.

A sentence production experiment was conducted to investigate F0 preplanning, specifically examining whether speakers plan pitch targets according to utterance length and whether preplanning limits dynamic control of pitch range. The experiment used a novel elicitation method in which visual stimuli that cue part of the utterance are delayed so that participants initiate the utterance without knowledge of its length. Participants read sentences in which the subject noun phrase was composed of one, two, or three conjoined noun phrases with controlled lexical and phonological content (e.g. Eight red weasels and nine green rhinos and eight blue llamas live in the zoo). On half of the trials with two and three noun phrases, the presentation of the visual stimuli that cued non-initial phrases (e.g., nine green rhinos and eight blue llamas) was delayed until after detection of utterance initiation. The effects of length and delay on F0 measures were analyzed. The results showed the length effect; the utterance-initial F0 peak was highest when three phrases were presented initially and lowest when one phrase was presented initially. Moreover, the utterance-initial peak in delayed stimuli was similar to that of the single phrase stimuli. In non-initial phrases, the pitch range of the delayed stimuli was constrained such that the F0 peak of the stimuli without delay was higher than those with delay. The findings suggest that speakers plan F0 targets and range before production, and such planning restricts subsequent control of F0.

Measurements of the physical outputs of speech—vocal tract geometry and acoustic energy—are high-dimensional, but linguistic theories posit a low-dimensional set of categories such as phonemes and phrase types. How can it be determined when and where in high-dimensional articulatory and acoustic signals there is information related to theoretical categories? For a variety of reasons, it is problematic to directly quantify mutual information between hypothesized categories and signals. To address this issue, a multi-scale analysis method is proposed for localizing category-related information in an ensemble of speech signals using machine learning algorithms. By analyzing how classification accuracy on unseen data varies as the temporal extent of training input is systematically restricted, inferences can be drawn regarding the temporal distribution of category-related information. The method can also be used to investigate redundancy between subsets of signal dimensions. Two types of theoretical categories are examined in this paper: phonemic/gestural categories and syntactic relative clause categories. Moreover, two different machine learning algorithms were examined: linear discriminant analysis and neural networks with long short-term memory units. Both algorithms detected category-related information earlier and later in signals than would be expected given standard theoretical assumptions about when linguistic categories should influence speech. The neural network algorithm was able to identify category-related information to a greater extent than the discriminant analyses.