Han-Gyol Yi

Significance: Speech sound learning in adulthood is a highly dynamic process. Here, we used direct neurophysiology of the human brain to examine learning-associated changes in neural activity with unprecedented spatiotemporal detail. While native English listeners were trained to identify unfamiliar pitch patterns in Mandarin, local neural populations throughout the cortex showed a diverse set of encoding properties for Mandarin sounds that tracked behavioral performance. While previous neuroimaging studies have focused on highlighting general differences across broad cortical regions, we demonstrate that these functionally heterogeneous populations are spatially interspersed with one another. These findings provide insight into how the human brain strikes a balance between stability and plasticity during learning in adulthood.

Introduction: Scalp‐recorded electrophysiological responses to complex, periodic auditory signals reflect phase‐locked activity from neural ensembles within the auditory system. These responses, referred to as frequency‐following responses (FFRs), have been widely utilized to index typical and atypical representation of speech signals in the auditory system. One of the major limitations in FFR is the low signal‐to‐noise ratio at the level of single trials. For this reason, the analysis relies on averaging across thousands of trials. The ability to examine the quality of single‐trial FFRs will allow investigation of trial‐by‐trial dynamics of the FFR, which has been impossible due to the averaging approach.Methods: In a novel, data‐driven approach, we used machine learning principles to decode information related to the speech signal from single trial FFRs. FFRs were collected from participants while they listened to two vowels produced by two speakers. Scalp‐recorded electrophysiological responses were projected onto a low‐dimensional spectral feature space independently derived from the same two vowels produced by 40 speakers, which were not presented to the participants. A novel supervised machine learning classifier was trained to discriminate vowel tokens on a subset of FFRs from each participant, and tested on the remaining subset.Results: We demonstrate reliable decoding of speech signals at the level of single‐trials by decomposing the raw FFR based on information‐bearing spectral features in the speech signal that were independently derived.Conclusions: Taken together, the ability to extract interpretable features at the level of single‐trials in a data‐driven manner offers unchartered possibilities in the noninvasive assessment of human auditory function.

We conducted a cross-linguistic study to evaluate the impact of language experience on midbrain encoding of acoustic dimensions. Midbrain electrophysiological responses were recorded to the four Mandarin tones in native Chinese (N = 10) and English (N = 10) listeners, through a counter-balanced block design. English participants were trained over multiple days to achieve tone categorization accuracy and reaction time equal to that of the Chinese participants. We assessed the extent to which the four Mandarin tones could be discerned from the electrophysiological responses, using a data-driven machine learning approach. The machine learning output was used to generate dissimilarity matrices that were subjected to a multidimensional scaling(MDS) model. A two dimensional MDS solution emerged that corresponded to “pitch height” and “pitch direction” of the Mandarin tones. Findings derived from the individual differences scaling (INDSCAL) method revealed that, initially, pitch direction was weighted more by the Chinese participants relative to the English participants. However, following training, relative weighting on pitch direction was found to increase in the English participants, comparable to the Chinese participants. These results suggest that long-term language experience and short-term training with Mandarin tones selectively enhances weighting of pitch dimensions at the level of the auditory midbrain.

A recent model suggests two strategies are involved in speech category learning: a reflective strategy that maps sounds onto categorical representations explicitly, and a reflexive strategy that does so implicitly. Successful learners transition from an initial reflective strategy to a reflexive strategy. We examined the extent to which speech category learning in adults can be improved by selectively enhancing these strategies in an optimal sequence, and how this influences performance in a lexical learning task. Monolingual English speakers (N = 40) learned to categorize Mandarin tones. Participants in the “optimal” condition (n = 10) were first presented with a combination of feedback information and talker presentation designed to enhance reflective strategies, and later presented with a feedback-talker combination that targeted reflexive strategies. This order was reversed in the “sub-optimal” condition (n = 10). Two control conditions (n = 10 each) exclusively targeted either strategy. All participants then learned 24 pseudo-Mandarin lexical items across three days. Participants in the optimal condition outperformed those in the sub-optimal and control conditions on the lexical learning task despite the fact that they did not attain the highest speech categorization accuracy. These results suggest that transitioning from reflective to reflexive strategies is ideal for applying learned speech categories to novel words.

Nonnative speech poses a challenge to speech perception, especially in challenging listening environments. Audiovisual (AV) cues are known to improve native speech perception in noise. The extent to which AV cues benefit nonnative speech perception in noise, however, is much less well-understood. Here, we examined native American English-speaking and native Korean-speaking listeners' perception of English sentences produced by a native American English speaker and a native Korean speaker across a range of signal-to-noise ratios (SNRs;−4 to −20 dB) in audio-only and audiovisual conditions. We employed psychometric function analyses to characterize the pattern of AV benefit across SNRs. For native English speech, the largest AV benefit occurred at intermediate SNR (i.e. −12 dB); but for nonnative English speech, the largest AV benefit occurred at a higher SNR (−4 dB). The psychometric function analyses demonstrated that the AV benefit patterns were different between native and nonnative English speech. The nativeness of the listener exerted negligible effects on the AV benefit across SNRs. However, the nonnative listeners' ability to gain AV benefit in native English speech was related to their proficiency in English. These findings suggest that the native language background of both the speaker and listener clearly modulate the optimal use of AV cues in speech recognition.

One of the most difficult category learning problems for humans is learning nonnative speech categories. While feedback-based category training can enhance speech learning, the mechanisms underlying these benefits are unclear. In this functional magnetic resonance imaging study, we investigated neural and computational mechanisms underlying feedback-dependent speech category learning in adults. Positive feedback activated a large corticostriatal network including the dorsolateral prefrontal cortex, inferior parietal lobule, middle temporal gyrus, caudate, putamen, and the ventral striatum. Successful learning was contingent upon the activity of domain-general category learning systems: the fast-learning reflective system, involving the dorsolateral prefrontal cortex that develops and tests explicit rules based on the feedback content, and the slow-learning reflexive system, involving the putamen in which the stimuli are implicitly associated with category responses based on the reward value in feedback. Computational modeling of response strategies revealed significant use of reflective strategies early in training and greater use of reflexive strategies later in training. Reflexive strategy use was associated with increased activation in the putamen. Our results demonstrate a critical role for the reflexive corticostriatal learning system as a function of response strategy and proficiency during speech category learning. Keywords: category learning, fMRI, corticostriatal systems, speech, putamen

Foreign-accented speech often presents a challenging listening condition. In addition to deviations from the target speech norms related to the inexperience of the nonnative speaker, listener characteristics may play a role in determining intelligibility levels. We have previously shown that an implicit visual bias for associating East Asian faces and foreignness predicts the listeners’ perceptual ability to process Korean-accented English audiovisual speech (Yi et al., 2013). Here, we examine the neural mechanism underlying the influence of listener bias to foreign faces on speech perception. In a functional magnetic resonance imaging (fMRI) study, native English speakers listened to native- and Korean-accented English sentences, with or without faces. The participants’ Asian-foreign association was measured using an implicit association test (IAT), conducted outside the scanner. We found that foreign-accented speech evoked greater activity in the bilateral primary auditory cortices and the inferior frontal gyri, potentially reflecting greater computational demand. Higher IAT scores, indicating greater bias, were associated with increased BOLD response to foreign-accented speech with faces in the primary auditory cortex, the early node for spectrotemporal analysis. We conclude the following: (1) foreign-accented speech perception places greater demand on the neural systems underlying speech perception; (2) face of the talker can exaggerate the perceived foreignness of foreign-accented speech; (3) implicit Asian-foreign association is associated with decreased neural efficiency in early spectrotemporal processing.