Seana Coulson
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Seana Coulson has a total of 86 co-authors. Here are the top co-authors by number of shared publications:
- David Brang
Neuroscience, Northwestern University
9 shared publications Marta Kutas
A Department of Cognitive Science , University of California , San Diego , CA , USA
8 shared publicationsUrsina Teuscher
Department of Behavioral Neuroscience, Oregon Health and Science University, Oregon, USA
6 shared publicationsYing Choon Wu
Swartz Center for Computational Neuroscience, UC San Diego, USA
6 shared publicationsTodd Oakley
5 shared publications- Ying Choon Wu
Center for Research in Language, UC San Diego 0526, 9500 Gilman Dr., La Jolla, CA 92093, USA; Swartz Center for Computational Neuroscience, UC San Diego 0559, 9500 Gilman Dr., La Jolla, CA 92093, USA
5 shared publications
Latest Publications
- Open Access
- 27 August 2026
- Published by Center for Open Science in PsyArXiv
Individual differences in visual imagery vividness predict neural responses to language-induced visual expectations
Grounded accounts of meaning propose that language comprehension recruits sensorimotor systems to represent described content. However, people differ markedly in how vividly they can picture objects and scenes in the mind's eye, raising the question of whether perceptual simulation is recruited similarly across individuals. We addressed this question using the sentence-picture verification task, in which pictures are verified more quickly when their shape matches rather than mismatches the state implied by a preceding sentence, a match effect taken to index perceptual simulation. In preregistered Experiment 1 (N = 110), matching pictures were verified faster and more accurately than mismatching pictures, yet the magnitude of this effect did not vary with imagery vividness. We reasoned that similar behavior might arise from different processes, so in Experiment 2 we recorded picture-locked event-related potentials (ERPs) from 34 participants performing the verification task. Shape mismatch elicited an early right-posterior response at 150-190 ms and a later, broader posterior response at 300-500 ms. Crucially, imagery vividness interacted with the shape match effect in the early window only: stronger imagers showed a pronounced right-posterior negativity, whereas verbal thinking tendency did not, indicating specificity to visual imagery. Together, results suggest that comparable behavioral outcomes may arise through partly different paths to meaning: an early, visually specific route recruited more strongly by vivid imagers, and a later conceptual route shared more broadly across the imagery spectrum. Language comprehension may draw on both perceptual simulation and linguistic associations, with the relative contribution of each depending in part on the comprehender.
- 27 August 2026
- Published by Center for Open Science in PsyArXiv
Open AccessIndividual differences in visual imagery vividness predict neural responses to language-induced visual expectations
Grounded accounts of meaning propose that language comprehension recruits sensorimotor systems to represent described content. However, people differ markedly in how vividly they can picture objects and scenes in the mind's eye, raising the question of whether perceptual simulation is recruited similarly across individuals. We addressed this question using the sentence-picture verification task, in which pictures are verified more quickly when their shape matches rather than mismatches the state implied by a preceding sentence, a match effect taken to index perceptual simulation. In preregistered Experiment 1 (N = 110), matching pictures were verified faster and more accurately than mismatching pictures, yet the magnitude of this effect did not vary with imagery vividness. We reasoned that similar behavior might arise from different processes, so in Experiment 2 we recorded picture-locked event-related potentials (ERPs) from 34 participants performing the verification task. Shape mismatch elicited an early right-posterior response at 150-190 ms and a later, broader posterior response at 300-500 ms. Crucially, imagery vividness interacted with the shape match effect in the early window only: stronger imagers showed a pronounced right-posterior negativity, whereas verbal thinking tendency did not, indicating specificity to visual imagery. Together, results suggest that comparable behavioral outcomes may arise through partly different paths to meaning: an early, visually specific route recruited more strongly by vivid imagers, and a later conceptual route shared more broadly across the imagery spectrum. Language comprehension may draw on both perceptual simulation and linguistic associations, with the relative contribution of each depending in part on the comprehender.
- Open Access
- 25 March 2026
- Published by Elsevier in Brain and Language
- Vol. 276, 105745
The influence of co-speech gestures on speech envelope tracking
- 25 March 2026
- Published by Elsevier in Brain and Language
- Vol. 276, 105745
Open AccessThe influence of co-speech gestures on speech envelope tracking
- Open Access
- 17 January 2026
- Published by Oxford University Press (OUP) in Neuroscience of Consciousness
- Vol. 2026 (1)
From dots to faces: individual differences in visual imagery capacity predict the content of Ganzflicker-induced hallucinations
A rapidly alternating red and black display known as Ganzflicker induces visual hallucinations that reflect the generative capacity of the visual system. Individuals vary in their degree of visual imagery, ranging from absent to vivid imagery. Recent proposals suggest that differences in the visual system along this imagery spectrum should also influence the complexity of other internally generated visual experiences. Here, we used tools from natural language processing to analyze free-text descriptions of hallucinations from >4000 participants, asking whether people with different imagery phenotypes see different things in their mind’s eye during Ganzflicker-induced hallucinations. Topic modeling of descriptions revealed that strong imagers described complex, naturalistic content, while weak imagers reported simple geometric patterns. Using crowd-sourced sensorimotor norms, we also found that participants with stronger imagery used language with richer perceptual associations. These findings may reflect individual variation in coordination between early visual areas and higher-order regions relevant for the imagery spectrum.
- 17 January 2026
- Published by Oxford University Press (OUP) in Neuroscience of Consciousness
- Vol. 2026 (1)
Open AccessFrom dots to faces: individual differences in visual imagery capacity predict the content of Ganzflicker-induced hallucinations
A rapidly alternating red and black display known as Ganzflicker induces visual hallucinations that reflect the generative capacity of the visual system. Individuals vary in their degree of visual imagery, ranging from absent to vivid imagery. Recent proposals suggest that differences in the visual system along this imagery spectrum should also influence the complexity of other internally generated visual experiences. Here, we used tools from natural language processing to analyze free-text descriptions of hallucinations from >4000 participants, asking whether people with different imagery phenotypes see different things in their mind’s eye during Ganzflicker-induced hallucinations. Topic modeling of descriptions revealed that strong imagers described complex, naturalistic content, while weak imagers reported simple geometric patterns. Using crowd-sourced sensorimotor norms, we also found that participants with stronger imagery used language with richer perceptual associations. These findings may reflect individual variation in coordination between early visual areas and higher-order regions relevant for the imagery spectrum.
- Open Access
Words and Worlds Both: Dynamic Effects of Distributional and Sensorimotor Information in Semantic Processing
An important issue in the semantic memory literature concerns the relative importance of experience-based sensorimotor versus language corpus-based distributional information in conceptual representations. To explore how each contributes to behavioral and neural responses on a conceptual task, EEG and RTs were recorded as healthy young adults viewed terms for concepts (e.g., “APPLE”) followed by properties (e.g., “red”) and pressed a button to indicate whether the property is true or false for the concept. Next, we constructed a series of mixed effects models of response times (RTs) and single-trial electroencephalogram (EEG) responses to the property words. Distributional models predicted data using semantic distance measures (e.g., between “APPLE” and “red”) derived from language corpus-based measures developed by computational linguists. Sensorimotor models predicted data using sensorimotor distance, a measure based on comparisons of each word’s experiential strength on the perceptual and action-effector dimensions from the crowd-sourced Lancaster Sensorimotor Norms. Statistical model comparison was used to determine whether the data was best fit by Distributional, Sensorimotor, or both sorts of information. In keeping with hybrid accounts of semantic memory, we find that both measures of semantic distance explained unique variance for behavioral and neural measures. Modelling EEG across seven successive 100-ms intervals revealed that the predictors’ temporal dynamics varies between true (APPLE – red) and false (APPLE – black) trials, but showed early sensorimotor activation for both. Results show how linguistic context and task demands modulate the recruitment of different information sources, supporting dynamic hybrid accounts of semantic memory.
Open AccessWords and Worlds Both: Dynamic Effects of Distributional and Sensorimotor Information in Semantic Processing
An important issue in the semantic memory literature concerns the relative importance of experience-based sensorimotor versus language corpus-based distributional information in conceptual representations. To explore how each contributes to behavioral and neural responses on a conceptual task, EEG and RTs were recorded as healthy young adults viewed terms for concepts (e.g., “APPLE”) followed by properties (e.g., “red”) and pressed a button to indicate whether the property is true or false for the concept. Next, we constructed a series of mixed effects models of response times (RTs) and single-trial electroencephalogram (EEG) responses to the property words. Distributional models predicted data using semantic distance measures (e.g., between “APPLE” and “red”) derived from language corpus-based measures developed by computational linguists. Sensorimotor models predicted data using sensorimotor distance, a measure based on comparisons of each word’s experiential strength on the perceptual and action-effector dimensions from the crowd-sourced Lancaster Sensorimotor Norms. Statistical model comparison was used to determine whether the data was best fit by Distributional, Sensorimotor, or both sorts of information. In keeping with hybrid accounts of semantic memory, we find that both measures of semantic distance explained unique variance for behavioral and neural measures. Modelling EEG across seven successive 100-ms intervals revealed that the predictors’ temporal dynamics varies between true (APPLE – red) and false (APPLE – black) trials, but showed early sensorimotor activation for both. Results show how linguistic context and task demands modulate the recruitment of different information sources, supporting dynamic hybrid accounts of semantic memory.
- Open Access
- 1 December 2025
- Published by MIT Press in Journal of Cognitive Neuroscience
- Vol. 37 (12) , 2616-2631
Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics
While multisensory super-additivity has been demonstrated in the context of visual articulation, it is unclear whether speech and co-speech gestures are similarly subject to super-additive integration. The current study investigates multisensory integration of speech and bodily gestures, testing whether biological motion signatures of co-speech gestures enhance cortical tracking of the speech envelope. We recorded EEG from 20 healthy adults as they watched a series of audiovisual (AV) discourse clips from four conditions: AV congruent clips with co-speech gestures that were naturally aligned with speech, AV incongruent clips in which gestures were not aligned with the speech, audio-only clips in which speech was delivered in isolation, and video-only clips presenting the gesture content with no accompanying speech. As we hypothesize that the kinematics of co-speech gestures are sufficient to drive gestural enhancement of speech tracking, our clips employed minimalistic “point-light” depictions of a speaker's movements: point-light talkers. Using neural decoder models to predict the amplitude of the speech envelope from EEG elicited in all four conditions, we compared speech reconstruction performance between multisensory (AV congruent) and additive models, that is, those representing the summed neural response across the two unisensory conditions. We found significant improvement in decoder scores for models trained on AV congruent trials relative to both audio-only and additive models. Forward models of brain activity indicated signatures of multisensory integration 140–160 msec following changes to the speech envelope. These results provide novel evidence for a multisensory enhancement effect of co-speech gesture kinematics on continuous speech tracking.
- 1 December 2025
- Published by MIT Press in Journal of Cognitive Neuroscience
- Vol. 37 (12) , 2616-2631
Open AccessPoint-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics
While multisensory super-additivity has been demonstrated in the context of visual articulation, it is unclear whether speech and co-speech gestures are similarly subject to super-additive integration. The current study investigates multisensory integration of speech and bodily gestures, testing whether biological motion signatures of co-speech gestures enhance cortical tracking of the speech envelope. We recorded EEG from 20 healthy adults as they watched a series of audiovisual (AV) discourse clips from four conditions: AV congruent clips with co-speech gestures that were naturally aligned with speech, AV incongruent clips in which gestures were not aligned with the speech, audio-only clips in which speech was delivered in isolation, and video-only clips presenting the gesture content with no accompanying speech. As we hypothesize that the kinematics of co-speech gestures are sufficient to drive gestural enhancement of speech tracking, our clips employed minimalistic “point-light” depictions of a speaker's movements: point-light talkers. Using neural decoder models to predict the amplitude of the speech envelope from EEG elicited in all four conditions, we compared speech reconstruction performance between multisensory (AV congruent) and additive models, that is, those representing the summed neural response across the two unisensory conditions. We found significant improvement in decoder scores for models trained on AV congruent trials relative to both audio-only and additive models. Forward models of brain activity indicated signatures of multisensory integration 140–160 msec following changes to the speech envelope. These results provide novel evidence for a multisensory enhancement effect of co-speech gesture kinematics on continuous speech tracking.
- Open Access
- 16 May 2025
- Published by Center for Open Science in PsyArXiv
Visual Imagery Vividness Predicts the Complexity of Induced Hallucinations
The current study utilizes the Ganzflicker paradigm—a flickering stimulus that induces visual hallucinations—to provide insight into the internally-generated visual experiences that correlate with individual differences in visual imagery. Here, we analyzed rich narrative descriptions of Ganzflicker hallucinations from 4,365 participants using natural language processing, sensorimotor norms, and AI visualizations. We find that overall perceptual richness and visual detail in descriptions increase with imagery vividness. Examining the specific content of these descriptions reveals that vivid imagers report more face and hand-related content than those with weaker imagery. Exploratory AI-generated visualizations of these descriptions provide additional insights, as those with weak imagery report patterns of simple visual features, like colors and geometric forms, while strong imagers’ hallucinations are filled with complex real-world stimuli. These findings suggest imagery differences may lie not in early visual processing but in the integration of basic visual features into complex object- and scene-level representations.
- 16 May 2025
- Published by Center for Open Science in PsyArXiv
Open AccessVisual Imagery Vividness Predicts the Complexity of Induced Hallucinations
The current study utilizes the Ganzflicker paradigm—a flickering stimulus that induces visual hallucinations—to provide insight into the internally-generated visual experiences that correlate with individual differences in visual imagery. Here, we analyzed rich narrative descriptions of Ganzflicker hallucinations from 4,365 participants using natural language processing, sensorimotor norms, and AI visualizations. We find that overall perceptual richness and visual detail in descriptions increase with imagery vividness. Examining the specific content of these descriptions reveals that vivid imagers report more face and hand-related content than those with weaker imagery. Exploratory AI-generated visualizations of these descriptions provide additional insights, as those with weak imagery report patterns of simple visual features, like colors and geometric forms, while strong imagers’ hallucinations are filled with complex real-world stimuli. These findings suggest imagery differences may lie not in early visual processing but in the integration of basic visual features into complex object- and scene-level representations.
- Open Access
Decoding Prosodic Information from Motion Capture Data: The Gravity of Co-Speech Gestures
In part due to correspondence in time, seeing how a speaking body moves can impact how speech is apprehended. Despite this, little is known about whether and which specific kinematic features of co-speech movements are relevant for their integration with speech. The current study uses machine learning techniques to investigate how co-speech gestures can be quantified to model vocal acoustics within an individual speaker. Specifically, we address whether kinetic descriptions of human movement are relevant for modeling their relationship with speech in time. To test this, we apply experimental manipulations that either highlight or obscure the relationship between co-speech movement kinematics and downward gravitational acceleration. Across two experiments, we provide evidence that quantifying co-speech movement as a function of its anisotropic relation to downward gravitational forces improves how well those co-speech movements can be used to predict prosodic dimensions of speech, as represented by the low-pass envelope. This study supports theoretical perspectives that invoke biomechanics to help explain speech-gesture synchrony and offers motivation for further behavioral or neuroimaging work investigating audiovisual integration and/or biological motion perception in the context of multimodal discourse.
Open AccessDecoding Prosodic Information from Motion Capture Data: The Gravity of Co-Speech Gestures
In part due to correspondence in time, seeing how a speaking body moves can impact how speech is apprehended. Despite this, little is known about whether and which specific kinematic features of co-speech movements are relevant for their integration with speech. The current study uses machine learning techniques to investigate how co-speech gestures can be quantified to model vocal acoustics within an individual speaker. Specifically, we address whether kinetic descriptions of human movement are relevant for modeling their relationship with speech in time. To test this, we apply experimental manipulations that either highlight or obscure the relationship between co-speech movement kinematics and downward gravitational acceleration. Across two experiments, we provide evidence that quantifying co-speech movement as a function of its anisotropic relation to downward gravitational forces improves how well those co-speech movements can be used to predict prosodic dimensions of speech, as represented by the low-pass envelope. This study supports theoretical perspectives that invoke biomechanics to help explain speech-gesture synchrony and offers motivation for further behavioral or neuroimaging work investigating audiovisual integration and/or biological motion perception in the context of multimodal discourse.
- Top Cited PapersOpen Access
- 1 April 2024
- Published by MIT Press in Neurobiology of Language
- Vol. 5 (1) , 107-135
Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects
Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.
- 1 April 2024
- Published by MIT Press in Neurobiology of Language
- Vol. 5 (1) , 107-135
Top Cited PapersOpen AccessStrong Prediction: Language Model Surprisal Explains Multiple N400 Effects
Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.
- Open Access
- 13 December 2023
- Published by Center for Open Science in PsyArXiv
Turing Jest: Do Large Language Models have a Sense of Humor?
Humor is an essential aspect of human experience, yet surprisingly little is known about how we recognize and understand humorous utterances. Most theories emphasize the role of incongruity detection and resolution, as well as cognitive capacities like Theory of Mind or pragmatic reasoning. In multiple pre-registered experiments, we ask whether the ability to understand verbal humor can emerge from exposure to purely linguistic input. We find that GPT-3, a large language model (LLM) trained on only language data, exhibits above-chance performance in tasks designed to detect, appreciate, and comprehend jokes. Although GPT-3 falls short of human performance, both humans and LLMs misclassify non-jokes with surprising endings as jokes. Further exploratory analyses reveal a relationship between model size and humor comprehension ability. Results suggest first, that LLMs are surprisingly adept at humor comprehension, and second, that language is not all one needs to “get the joke”.
- 13 December 2023
- Published by Center for Open Science in PsyArXiv
Open AccessTuring Jest: Do Large Language Models have a Sense of Humor?
Humor is an essential aspect of human experience, yet surprisingly little is known about how we recognize and understand humorous utterances. Most theories emphasize the role of incongruity detection and resolution, as well as cognitive capacities like Theory of Mind or pragmatic reasoning. In multiple pre-registered experiments, we ask whether the ability to understand verbal humor can emerge from exposure to purely linguistic input. We find that GPT-3, a large language model (LLM) trained on only language data, exhibits above-chance performance in tasks designed to detect, appreciate, and comprehend jokes. Although GPT-3 falls short of human performance, both humans and LLMs misclassify non-jokes with surprising endings as jokes. Further exploratory analyses reveal a relationship between model size and humor comprehension ability. Results suggest first, that LLMs are surprisingly adept at humor comprehension, and second, that language is not all one needs to “get the joke”.
- Open Access
- 23 July 2023
- Published by Taylor & Francis in Language, Cognition and Neuroscience
- Vol. 38 (10) , 1531-1553
Moving thoughts: emotion concepts from the perspective of context dependent embodied simulation
- 23 July 2023
- Published by Taylor & Francis in Language, Cognition and Neuroscience
- Vol. 38 (10) , 1531-1553
Open AccessMoving thoughts: emotion concepts from the perspective of context dependent embodied simulation