Michael S. Beauchamp

This interview brings together perspectives from the authors on the origins, evolution, and future directions of multisensory neuroscience. The authors reflect on their early motivations, tracing the field’s development from studies of intersensory attention and early cortical integration to contemporary investigations of naturalistic audiovisual speech. They highlight landmark empirical and methodological advances — including high-density EEG, BOLD fMRI, intracranial recordings, and analyses of continuous stimuli — that overturned the traditional view of multisensory integration as a strictly higher-order process. Central themes include the role of the superior temporal sulcus in audiovisual speech, the hierarchical and temporally dynamic nature of multisensory binding, and ongoing debates concerning the mechanisms underlying neural responses to naturalistic stimuli, such as evoked-response versus oscillatory-entrainment accounts. The interview also addresses current controversies, including interpretations of redundant signals effects and the neural organization of language, as well as concerns about underpowered studies and reproducibility. The authors strongly advocate open science practices, emphasizing data and code sharing while acknowledging structural challenges. Looking ahead, they discuss the transformative potential of artificial intelligence, invasive human recordings, genetic models, and translational applications, particularly for neurodevelopmental disorders. Collectively, this dialogue situates multisensory integration as a rapidly advancing field bridging sensory processing, cognition, and clinical relevance.

Seeing the face of the talker improves the intelligibility of speech-in-noise, termed the visualbenefit. The visual benefit is usually measured at the whole-word level, with responses classifiedas "correct" or "incorrect". An alternative method measures visual benefit using the phonemeoverlap between stimulus and response. In 55 participants presented with auditory andaudiovisual words, phoneme-overlap analysis estimated 289 different visual benefit values,compared with only three different values for whole-word analysis. Phoneme-overlap detected apositive visual benefit in 67% of word-participant pairs, in contrast to 32% for whole-wordanalysis. Individual words varied greatly in their phoneme-overlap visual benefit, from 1% to72%. Variability across words was accurately predicted (r = 0.78) by the phoneme compositionof the word and the visual benefit for untested words could be predicted with a median error of9%. Measuring the visual benefit for individual phonemes revealed similarly high variability(1% to 70%) with six phonemes (W, F, V, TH, P, AW) exhibiting visual benefit more than onestandard deviation above the mean. The increased sensitivity of phoneme-overlap analysisrelative to conventional whole-word analysis may provide a useful tool for investigatingaudiovisual speech perception in clinical, neuroscience and computer science contexts.

Miniaturized neural interfaces for research, diagnostics, and neuromodulation therapies require electrode materials that maintain low impedance and high charge injection capacity as device dimensions shrink to ensure high‐quality recordings and safe stimulation. Conventional interfaces rely on metals like platinum (Pt), which are limited by intrinsically high impedance and low charge transfer capacity, reducing their performance in sub‐100 µm applications. Ti ₃ C ₂ T _x MXene has emerged as a promising alternative for high‐density recording and stimulation interfaces, though the fundamental charge transfer mechanisms governing its performance remain poorly understood. This study evaluates Ti ₃ C ₂ T _x MXene microelectrodes across a range of diameters (25–500 µm) and systematically elucidates the mechanisms governing their recording and stimulation capabilities. Electrochemical impedance spectroscopy, cyclic voltammetry, and voltage transient measurements—supported by equivalent‐circuit modeling—revealed enhanced recording and stimulation capabilities of the MXene microelectrodes over size‐matched Pt microelectrodes, attributed to reduced charge‐transfer resistance and increased double‐layer capacitance. Finally, varying the volume and concentration of the spray‐coated Ti ₃ C ₂ T _x films showed that increased MXene concentration and volume enhanced performance by creating thicker, rougher interfaces. Together, these results establish Ti ₃ C ₂ T _x MXene as a promising electrode material with exceptionally low impedance, high capacitance, and high charge‐injection capacity across the microscale.

Precise and synchronized multimodal data capture in neurosurgical environments is essential for further understanding brain function and will be crucial to advancing the development of brain-computer interface technology. We have developed an open-source software platform named Thalamus, for multimodal data capture integrated with existing sensors and hardware commonly utilized in the operating room and other clinical environments such as pulse oximeters, inertial sensors, electromyography and neural electrophysiology. Thalamus facilitates synchronous recording of neural and behavioral data, enabling real-time computation for closed-loop experiments and detailed analysis of complex motor functions and neural activity. Thalamus uses a modular, configurable node-based pipeline with a tiered Python and C + + architecture. These design elements allow Thalamus to support a wide range of high-resolution sensors for diverse behavioral data types and enable robust closed-loop synchronization of various data streams. Validation experiments demonstrate that Thalamus is capable of data integration and concurrent analysis with up to sub-millisecond precision, offering great potential for enhancing neurosurgical research and clinical applications. By leveraging conventional sensors and hardware already in use, Thalamus supports adoption into the clinical environment, paving the way for more comprehensive, data-driven approaches to neurological care and improving the personalization and rigor of treatment strategies.

Miniaturized neural interfaces for research, diagnostics, and neuromodulation therapies require electrode materials that maintain low impedance and high charge injection capacity as device dimensions shrink to ensure high-quality recordings and safe stimulation. Conventional interfaces rely on metals like platinum (Pt), which are limited by intrinsically high impedance and low charge transfer capacity, reducing their performance in sub-100 µm applications. Ti ₃ C ₂ T _x MXene has emerged as a promising alternative for high-density recording and stimulation interfaces, though the fundamental charge transfer mechanisms governing its performance remain poorly understood. This study evaluates Ti ₃ C ₂ T _x MXene microelectrodes across a range of diameters (25 - 500 µm) and systematically elucidates the mechanisms governing their recording and stimulation capabilities. Electrochemical impedance spectroscopy, cyclic voltammetry, and voltage transient measurements – supported by equivalent-circuit modeling – revealed enhanced recording and stimulation capabilities of the MXene microelectrodes over size-matched Pt microelectrodes, attributed to reduced charge-transfer resistance and increased double-layer capacitance. Finally, varying the volume and concentration of the spray-coated Ti ₃ C ₂ T _x films showed that increased MXene concentration and volume enhanced performance by creating thicker, rougher interfaces. Together, these results establish Ti ₃ C ₂ T _x MXene as a promising electrode material with exceptional performance at the microscale.

In the McGurk effect, perception of an auditory syllable changes dramatically when it is paired with an incongruent visual syllable, countering our intuition that speech perception is solely an auditory process. The dominant modeling framework for the study of audiovisual speech perception is that of Bayesian causal inference, but current Bayesian models are unable to predict the wide range of percepts evoked by McGurk syllables. We explored whether a deep neural network (DNN) known as AVHuBERT could provide an alternative modeling framework. AVHuBERT model variants were presented with McGurk syllables consisting of auditory “ba” paired with visual “ga” recorded from eight different talkers. AVHuBERT identified McGurk syllables as something other than “ba” at a rate of 59%, demonstrating a robust McGurk effect. The rate of the McGurk effect was similar to that observed in humans: 100 participants presented with the same McGurk syllables reported non-“ba” percepts on 56% of trials. AVHuBERT variants and humans produced a wide variety of responses to McGurk syllables, including the canonical McGurk fusion percept of “da,” responses without any initial consonant such as “ah” and responses with other initial consonants such as “fa.” The ability to predict percepts experienced by humans but not predicted by current Bayesian models suggest that DNNs and Bayesian models may provide complementary windows into the perceptual mechanisms underlying human audiovisual speech perception.

Accurate identification of the seizure onset zone (SOZ) using intracranial electroencephalography (iEEG) remains challenging. Although diverse methods have leveraged spectral features to classify patient outcomes, few approaches focus on identifying individual electrodes within the SOZ or integrate a broad spectrum of frequency ranges within a single model. We developed an interpretable machine learning model that integrates power across delta, theta, alpha, beta, gamma and high-gamma frequencies over time to identify the SOZ. For 1511 electrodes implanted across 21 patients, we computed the mean spectral power in each frequency band for the first 20 s after seizure onset and analysed the differences in power between SOZ and non-SOZ electrodes. In patients who were seizure free after surgery (n = 14), electrodes within the SOZ showed significantly higher area under the curve (AUC) for mean power over time in the first 20 s after seizure onset compared to electrodes outside the SOZ in the alpha (P = 0.0272), beta (P = 0.0263), gamma (P = 0.0013) and high gamma (P = 0.0086) ranges. Additionally, electrodes within the SOZ in patients who became seizure free after surgery had significantly higher AUC compared to equivalent electrodes in patients who did not become seizure free after surgery (n = 7) in the gamma (P = 0.0145) and high gamma (P = 0.0024) power ranges. We trained a stacked random forest ensemble model using these features over time to label electrodes within the SOZ. Leave-one-out patient cross-validation of the machine learning model yielded a 96.6% positive predictive value and 99.9% specificity for identifying electrodes within the SOZ. Our dataset included a diverse array of seizure onset patterns, which were all classified accurately by the model. A second model was trained to predict post-operative seizure freedom, yielding 95.2% accuracy for predicting seizure outcome based on a planned resection. This two-model design mirrors clinical workflow, first localizing SOZ electrodes to support surgical planning and then predicting outcomes based on a surgical plan. An advantage of our interpretable machine learning approach is the ability to interrogate our models to understand how predictions are made. For electrode classification, the model weighed beta (0.66 ± 0.07), high gamma (0.54 ± 0.06) and delta (0.51 ± 0.06) power bands most heavily. Viewing the model’s frequency band weights over time reveals that the model identified a pattern resembling the ‘fingerprint of the epileptogenic zone’, reinforcing the importance of this dominant fundamental neurophysiologic signature of seizure onset.

Human speech perception is multisensory, integrating auditory information from the talker's voice with visual information from the talker's face. BOLD fMRI studies have implicated the superior temporal gyrus (STG) in processing auditory speech and the superior temporal sulcus (STS) in integrating auditory and visual speech, but as an indirect hemodynamic measure, fMRI is limited in its ability to track the rapid neural computations underlying speech perception. Using stereoelectroencephalography (sEEG) electrodes, we directly recorded from the STG and STS in 42 epilepsy patients (25F, 17M). Participants identified single words presented in auditory, visual, and audiovisual formats with and without added auditory noise. Seeing the talker's face provided a strong perceptual benefit, improving perception of noisy speech in every participant. Neurally, a subpopulation of electrodes concentrated in mid-posterior STG and STS responded to both auditory speech (latency 71 ms) and visual speech (109 ms). Significant multisensory enhancement was observed, especially in the upper bank of the STS: compared with auditory-only speech, the response latency for audiovisual speech was 40% faster and the response amplitude was 18% larger. In contrast, STG showed neither faster nor larger multisensory responses. Surprisingly, STS response latencies for audiovisual speech were significantly faster than those in the STG (50 ms vs 64 ms), suggesting a parallel pathway model in which the STG plays the primary role in auditory-only speech perception, while the STS takes the lead in audiovisual speech perception. Together with fMRI, sEEG provides converging evidence that STS plays a key role in multisensory integration.

In the McGurk effect, perception of an auditory syllable changes dramatically when it is paired with an incongruent visual syllable, countering our intuition that speech perception is solely an auditory process. The dominant modeling framework for the study of audiovisual speech perception is that of Bayesian causal inference, but current Bayesian models are unable to predict the wide range of percepts evoked by McGurk syllables. We explored whether a deep neural network (DNN) known as AVHuBERT could provide an alternative modeling framework. AVHuBERT model variants were presented with McGurk syllables consisting of auditory “ba” paired with visual “ga” recorded from 8 different talkers. AVHuBERT identified McGurk syllables as something other than “ba” at a rate of 59%, demonstrating a robust McGurk effect. The rate of the McGurk effect was similar to that observed in humans: 100 participants presented with the same McGurk syllables reported non-“ba” percepts on 56% of trials. AVHubert variants and humans produced a wide variety of responses to McGurk syllables, including the canonical McGurk fusion percept of “da“, responses without any initial consonant such as “ah” and responses with other initial consonants such as “fa“. The ability to predict percepts experienced by humans but not predicted by current Bayesian models suggest that DNNs and Bayesian models may provide complementary windows into the perceptual mechanisms underlying human audiovisual speech perception.

Brain responses to sensory stimuli depend on the specific task performed by the observer. Previously, fMRI measurements in ventral temporal cortex (VTC) showed that BOLD responses to words and faces scale with the difficulty of a categorization task compared to responses observed during a fixation task (Kay and Yeatman, 2017). This scaling of BOLD responses is thought to be driven by the engagement of cognitive functions during image categorization. To understand how these cognitive task demands change dynamically over time and influence neural activity in VTC, we measured electrocorticography (ECoG) data in two human participants during the same experimental tasks. The ECoG high frequency broadband activity (>70 Hz) showed that local neuronal responses increased with image contrast (as expected for sensory encoding) and were scaled by task demands 0.2 seconds after stimulus onset. In contrast, ECoG low frequency activity in the alpha/beta range (8-28 Hz) was insensitive to image contrast and showed larger suppression with increasing task demands. These results indicate that high frequencies in VTC reflect both the encoding of sensory inputs and task demands, similar to prior BOLD response measurements, whereas low frequency oscillations represent task demands, not sensory inputs per se. In line with the interpretation that low frequency oscillations reflect pulsed inhibition, we speculate that suppression of low frequency oscillations amplifies neural activity in VTC in support of visual task demands.