JFJustin T. Fleming

Justin T. Fleming

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Justin T. Fleming has a total of 23 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Visual Gestures of the Head and Eyebrows Support Prosody Perception for Individuals with Cochlear Implants

    by Justin T. Fleming, Harley J. Wheeler, Matthew B. Winn

    Visual gestures, especially head movements and eyebrow raises, are time-locked to acoustic cues during the expression of spoken prosody. The present study examined the role these visual cues play in prosody perception, particularly for individuals with cochlear implants (CIs), who often experience challenges understanding prosody due to reduced access to pitch cues. A vocal mimicry paradigm was used to obtain granular, objective measures of prosody perception through acoustic analysis of mimicked sentences. Stimuli consisted of audio-visual recordings from one talker that captured naturally occurring variability in the expression of auditory and visual cues to word focus. Participants mimicked these natural recordings, as well as prosody-transplanted stimuli that allowed us to isolate the influence of auditory or visual prosody cues while holding the other modality at a neutral level (broad focus). Participants converted visual prosody cues into acoustic correlates of prosody in their mimicry, repeating acoustically unfocused words with higher F0 and intensity when those words were paired with video containing head and eyebrow gestures. This visual influence was stronger for participants with CIs than an age-matched group of typical-hearing listeners. CI participants who were less successful at acoustic mimicry tended to be more influenced by visual cues. These results indicate that CI listeners compensate for degraded auditory cues by integrating visual gestures into their perception of spoken prosody, potentially highlighting new targets for multisensory counseling or training.


  • Open Access

    Seeing a Talker’s Mouth Reduces the Effort of Perceiving Speech and Repairing Perceptual Mistakes for Listeners With Cochlear Implants

    by Justin T. Fleming, Matthew B. Winn

    Objectives: Seeing a talker’s mouth improves speech intelligibility, particularly for listeners who use cochlear implants (CIs). However, the impacts of visual cues on listening effort for listeners with CIs remain poorly understood, as previous studies have focused on listeners with typical hearing (TH) and featured stimuli that do not invoke effortful cognitive speech perception challenges. This study directly compared the effort of perceiving audiovisual speech between listeners who use CIs and those with TH. Visual cues were hypothesized to yield more relief from listening effort in a cognitively challenging speech perception condition that required listeners to mentally repair a missing word in the auditory stimulus. Eye gaze was simultaneously measured to examine whether the tendency to look toward a talker’s mouth would increase during these moments of uncertainty about the speech stimulus. Design: Participants included listeners with CIs and an age-matched group of participants with typical age-adjusted hearing (N = 20 in both groups). The magnitude and time course of listening effort were evaluated using pupillometry. In half of the blocks, phonetic visual cues were severely degraded by selectively blurring the talker’s mouth, which preserved stimulus luminance so visual conditions could be compared using pupillometry. Each block included a mixture of trials in which the sentence audio was intact, and trials in which a target word in the auditory stimulus was replaced by noise; the latter required participants to mentally reconstruct the target word upon repeating the sentence. Pupil and gaze data were analyzed using generalized additive mixed-effects models to identify the stretches of time during which effort or gaze strategy differed between conditions. Results: Visual release from effort was greater and lasted longer for listeners with CIs compared with those with TH. Within the CI group, visual cues reduced effort to a greater extent when a missing word needed to be repaired than when the speech was intact. Seeing the talker’s mouth also improved speech intelligibility for listeners with CIs, including reducing the number of incoherent verbal responses when repair was required. The two hearing groups deployed different gaze strategies when perceiving audiovisual speech. CI listeners looked more at the mouth overall, even when it was blurred, while TH listeners tended to increase looks to the mouth in the moment following a missing word in the auditory stimulus. Conclusions: Integrating visual cues from a talker’s mouth not only improves speech intelligibility but also reduces listening effort, particularly for listeners with CIs. For listeners with CIs (but not those with TH), these visual benefits are magnified when a missed word needs to be mentally corrected—a common occurrence during everyday speech perception for individuals with hearing loss. These results underscore the importance of including participants with hearing loss in listening effort studies and suggest caution in assuming results from TH listeners will generalize to those with hearing loss. They also highlight the potential clinical relevance of visual speech information, for counseling patients and families and potentially for the development of audiovisual strategies to reduce listening effort.


  • Open Access

    Seeing a talker’s mouth reduces the effort of perceiving speech and repairing perceptual mistakes for listeners with cochlear implants

    by Justin Fleming, Matthew Winn

    Objectives: Seeing a talker’s mouth improves speech intelligibility, particularly for listeners who use cochlear implants (CIs). However, the impacts of visual cues on listening effort for listeners with CIs remain poorly understood, as previous studies have focused on listeners with typical hearing (TH) and featured stimuli that do not invoke effortful cognitive speech perception challenges. This study directly compared the effort of perceiving audiovisual speech between listeners who use CIs and those with TH. Visual cues were hypothesized to yield more relief from listening effort in a cognitively challenging speech perception condition that required listeners to mentally repair a missing word in the auditory stimulus. Eye gaze was simultaneously measured to examine whether the tendency to look toward a talker’s mouth would increase during these moments of uncertainty about the speech stimulus. Design: Participants included listeners with CIs and an age-matched group of participants with typical age-adjusted hearing (N = 20 in both groups). The magnitude and time course of listening effort was evaluated using pupillometry. In half of the blocks, phonetic visual cues were removed by selectively blurring the talker’s mouth, which preserved stimulus luminance so visual conditions could be compared using pupillometry. Each block included a mixture of trials in which the sentence audio was intact, and trials in which a target word in the auditory stimulus was replaced by noise; the latter required participants to mentally reconstruct the target word upon repeating the sentence. Pupil and gaze data were analyzed using generalized additive mixed effects models (GAMMs) to identify the stretches of time during which effort or gaze strategy differed between conditions.Results: Visual release from effort was greater and lasted longer for listeners with CIs compared to those with TH. Within the CI group, visual cues reduced effort to a greater extent when a missing word needed to be repaired than when the speech was intact. Seeing the talker’s mouth also improved speech intelligibility for listeners with CIs, including reducing the number of incoherent verbal responses when repair was required. The two hearing groups deployed different gaze strategies when perceiving audiovisual speech. CI listeners looked more at the mouth overall, even when it was blurred, while TH listeners tended to increase looks to the mouth in the moment following a missing word in the auditory stimulus. Conclusions: Integrating visual cues from a talker’s mouth not only improves speech intelligibility, but also reduces listening effort, particularly for listeners with CIs. For listeners with CIs (but not those with TH), these visual benefits are magnified when a missed word needs to be mentally corrected — a common occurrence during everyday speech perception for individuals with hearing loss. These results underscore the importance of including participants with hearing loss in listening effort studies and suggest caution in assuming results from TH listeners will generalize to those with hearing loss. They also highlight the potential clinical relevance of visual speech information, for counseling patients and families and potentially for the development of audiovisual strategies to reduce listening effort.

  • Open Access

    Sensory modality and information domain contribute jointly to dual-task interference between working memory and perceptual processing

    by Justin T. Fleming, J. Michelle Njoroge, Abigail L. Noyce, Tyler K. Perrachione, Barbara G. Shinn-Cunningham

    Making sense of our environment requires us to extract temporal and spatial information from multiple sensory modalities, particularly audition and vision. Often, we must hold this sensory information in working memory (WM) to guide future actions, while simultaneously processing new sensory inputs as they arise. However, these processes of WM maintenance and perceptual processing can interfere with one another when the tasks rely on similar cognitive resources. fMRI studies have uncovered attention and WM networks that are specialized for either auditory-temporal or visual-spatial processing; the functional specialization of these networks makes specific predictions about patterns of interference between perceptual processing and WM. Specifically, we hypothesized that dual-task interference should increase when the tasks share a common sensory modality, a common information domain (temporal vs. spatial processing), or both. To test these predictions, we asked participants to store temporal or spatial information about auditory or visual stimuli in WM. On some trials, participants also performed an intervening auditory task, which was either temporal or spatial, during WM retention. Errors on WM recall and perceptual judgment tasks both generally increased when the tasks relied on shared modality- and domain-biased resources, with maximal interference when both tasks were auditory-temporal. Pupil dilations were also larger and started earlier when both tasks were auditory-temporal, indicating an increase in cognitive effort to overcome the interference. Event-related potentials (ERPs) and alpha-band oscillatory activity revealed neural signatures of domain-based interference even when the tasks were presented in different sensory modalities, when behavioral differences were masked by ceiling effects. These results demonstrate that sensory modality and information domain jointly affect how task information is represented in WM, consistent with past work demonstrating how tasks engage complementary auditory-temporal and visual-spatial cognitive control networks.


  • Open Access

    Strategic perceptual weighting of acoustic cues for word stress in listeners with cochlear implants, acoustic hearing, or simulated bimodal hearing

    by Justin T. Fleming, Matthew B. Winn

    Perception of word stress is an important aspect of recognizing speech, guiding the listener toward candidate words based on the perceived stress pattern. Cochlear implant (CI) signal processing is likely to disrupt some of the available cues for word stress, particularly vowel quality and pitch contour changes. In this study, we used a cue weighting paradigm to investigate differences in stress cue weighting patterns between participants listening with CIs and those with normal hearing (NH). We found that participants with CIs gave less weight to frequency-based pitch and vowel quality cues than NH listeners but compensated by upweighting vowel duration and intensity cues. Nonetheless, CI listeners' stress judgments were also significantly influenced by vowel quality and pitch, and they modulated their usage of these cues depending on the specific word pair in a manner similar to NH participants. In a series of separate online experiments with NH listeners, we simulated aspects of bimodal hearing by combining low-pass filtered speech with a vocoded signal. In these conditions, participants upweighted pitch and vowel quality cues relative to a fully vocoded control condition, suggesting that bimodal listening holds promise for restoring the stress cue weighting patterns exhibited by listeners with NH.


  • Open Access

    Spatial alignment between faces and voices improves selective attention to audio-visual speech

    by Justin T. Fleming, Ross K. Maddox, Barbara G. Shinn-Cunningham

    The ability to see a talker's face improves speech intelligibility in noise, provided that the auditory and visual speech signals are approximately aligned in time. However, the importance of spatial alignment between corresponding faces and voices remains unresolved, particularly in multi-talker environments. In a series of online experiments, we investigated this using a task that required participants to selectively attend a target talker in noise while ignoring a distractor talker. In experiment 1, we found improved task performance when the talkers' faces were visible, but only when corresponding faces and voices were presented in the same hemifield (spatially aligned). In experiment 2, we tested for possible influences of eye position on this result. In auditory-only conditions, directing gaze toward the distractor voice reduced performance, but this effect could not fully explain the cost of audio-visual (AV) spatial misalignment. Lowering the signal-to-noise ratio (SNR) of the speech from +4 to −4 dB increased the magnitude of the AV spatial alignment effect (experiment 3), but accurate closed-set lipreading caused a floor effect that influenced results at lower SNRs (experiment 4). Taken together, these results demonstrate that spatial alignment between faces and voices contributes to the ability to selectively attend AV speech.


  • Open Access

    Patterns of information segregation during working memory and attention revealed by dual-task interference in behavior, pupillometry, and EEG

    by Justin T. Fleming, J. Michelle Njoroge, Abigail L. Noyce, Tyler K. Perrahione, Barbara G. Shinn-Cunningham

    Project Making sense of our environment requires us to extract simultaneous temporal and spatial information from multiple sensory modalities, particularly audition and vision. This sensory information can be stored in working memory (WM) to guide future actions, at which point it must be safeguarded against interference from ongoing sensory processing. Recent fMRI research has uncovered regions in human frontal cortex well-suited to coordinate this interplay between attention and WM for multisensory and multidimensional information. Which of these brain regions are engaged depends on both the sensory modality of the input and the information domain of the task, forming the basis of two complementary networks specialized for auditory/temporal and visual/spatial processing. Motivated by the functional specializations of these networks, we examined whether similarity in sensory modality and information domain modulates neural and perceptual interference between two concurrent tasks. Participants stored temporal or spatial information about auditory or visual stimuli in WM, and on some trials, performed an intervening temporal or spatial auditory task during WM retention. WM recall and auditory perceptual judgments were impaired when the two tasks relied on the same sensory modality and/or information domain. Pupil dilations were also larger in these conditions, indicating increased cognitive effort. Event-related potentials (ERPs) revealed a neural signature of domain-based interference that was masked by behavioral ceiling effects. These results demonstrate that modality and information domain jointly affect how task information is represented in WM, and concomitantly, how tasks engage the complementary auditory-temporal and visual/spatial cognitive control networks.


  • Open Access

    Spatial alignment between faces and voices improves selective attention to audio-visual speech

    by Justin T. Fleming, Ross K. Maddox, Barbara G. Shinn-Cunningham

    The ability to see a talker’s face has long been known to improve speech intelligibility in noise. This perceptual benefit depends on approximate temporal alignment between the auditory and visual speech components. However, the practical role that cross-modal spatial alignment plays in integrating audio-visual (AV) speech remains unresolved, particularly when competing talkers are present. In a series of online experiments, we investigated the importance of spatial alignment between corresponding faces and voices using a paradigm that featured both acoustic masking (speech-shaped noise) and attentional demands from a competing talker. Participants selectively attended a Target Talker’s speech, then identified a word spoken by the Target Talker. In Exp. 1, we found improved task performance when the talkers’ faces were visible, but only when corresponding faces and voices were presented in the same hemifield (spatially aligned). In Exp. 2, we tested for possible influences of eye position on this result. In auditory-only conditions, directing gaze toward the distractor voice reduced performance as predicted, but this effect could not fully explain the cost of AV spatial misalignment. Finally, in Exp. 3 and 4, we show that the effect of AV spatial alignment changes with noise level, but this was limited by a floor effect: due to the use of closed-set stimuli, participants were able to perform the task relatively well using lipreading alone. However, comparison between the results of Exp. 1 and Exp. 3 suggests that the cost of AV misalignment is larger at high noise levels. Overall, these results indicate that spatial alignment between corresponding faces and voices is important for AV speech integration in attentionally demanding communication settings.

  • Open Access

    Multiple time scales of the ventriloquism aftereffect

    by Adam K Bosen, Justin T Fleming, Paul D Allen, William E O'Neill, Gary D Paige

    The ventriloquism aftereffect (VAE) refers to a shift in auditory spatial perception following exposure to a spatial disparity between auditory and visual stimuli. The VAE has been previously measured on two distinct time scales. Hundreds or thousands of exposures to a an audio-visual spatial disparity produces enduring VAE that persists after exposure ceases. Exposure to a single audio-visual spatial disparity produces immediate VAE that decays over seconds. To determine if these phenomena are two extremes of a continuum or represent distinct processes, we conducted an experiment with normal hearing listeners that measured VAE in response to a repeated, constant audio-visual disparity sequence, both immediately after exposure to each audio-visual disparity and after the end of the sequence. In each experimental session, subjects were exposed to sequences of auditory and visual targets that were constantly offset by +8° or −8° in azimuth from one another, then localized auditory targets presented in isolation following each sequence. Eye position was controlled throughout the experiment, to avoid the effects of gaze on auditory localization. In contrast to other studies that did not control eye position, we found both a large shift in auditory perception that decayed rapidly after each AV disparity exposure, along with a gradual shift in auditory perception that grew over time and persisted after exposure to the AV disparity ceased. We modeled the temporal and spatial properties of the measured auditory shifts using grey box nonlinear system identification, and found that two models could explain the data equally well. In the power model, the temporal decay of the ventriloquism aftereffect was modeled with a power law relationship. This causes an initial rapid drop in auditory shift, followed by a long tail which accumulates with repeated exposure to audio-visual disparity. In the double exponential model, two separate processes were required to explain the data, one which accumulated and decayed exponentially and the other which slowly integrated over time. Both models fit the data best when the spatial spread of the ventriloquism aftereffect was limited to a window around the location of the audio-visual disparity. We directly compare the predictions made by each model, and suggest additional measurements that could help distinguish which model best describes the mechanisms underlying the VAE.


  • Accumulation and decay of visual capture and the ventriloquism aftereffect caused by brief audio-visual disparities

    by Adam K. Bosen, Justin T. Fleming, Paul D. Allen, William E. O‘nEill, Gary D. Paige

    Visual capture and the ventriloquism aftereffect resolve spatial disparities of incongruent auditory visual (AV) objects by shifting auditory spatial perception to align with vision. Here, we demonstrated the distinct temporal characteristics of visual capture and the ventriloquism aftereffect in response to brief AV disparities. In a set of experiments, subjects localized either the auditory component of AV targets (A within AV) or a second sound presented at varying delays (1–20 s) after AV exposure (A2 after AV). AV targets were trains of brief presentations (1 or 20), covering a ±30° azimuthal range, and with ±8° (R or L) disparity. We found that the magnitude of visual capture generally reached its peak within a single AV pair and did not dissipate with time, while the ventriloquism aftereffect accumulated with repetitions of AV pairs and dissipated with time. Additionally, the magnitude of the auditory shift induced by each phenomenon was uncorrelated across listeners and visual capture was unaffected by subsequent auditory targets, indicating that visual capture and the ventriloquism aftereffect are separate mechanisms with distinct effects on auditory spatial perception. Our results indicate that visual capture is a ‘sample-and-hold’ process that binds related objects and stores the combined percept in memory, whereas the ventriloquism aftereffect is a ‘leaky integrator’ process that accumulates with experience and decays with time to compensate for cross-modal disparities.