SCSeana Coulson

Seana Coulson

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Seana Coulson has a total of 86 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Individual differences in visual imagery vividness predict neural responses to language-induced visual expectations

    by Ana Chkhaidze, Lera Boroditsky, Sean Trott, Anastasia Kiyonaga, Seana Coulson

    Grounded accounts of meaning propose that language comprehension recruits sensorimotor systems to represent described content. However, people differ markedly in how vividly they can picture objects and scenes in the mind's eye, raising the question of whether perceptual simulation is recruited similarly across individuals. We addressed this question using the sentence-picture verification task, in which pictures are verified more quickly when their shape matches rather than mismatches the state implied by a preceding sentence, a match effect taken to index perceptual simulation. In preregistered Experiment 1 (N = 110), matching pictures were verified faster and more accurately than mismatching pictures, yet the magnitude of this effect did not vary with imagery vividness. We reasoned that similar behavior might arise from different processes, so in Experiment 2 we recorded picture-locked event-related potentials (ERPs) from 34 participants performing the verification task. Shape mismatch elicited an early right-posterior response at 150-190 ms and a later, broader posterior response at 300-500 ms. Crucially, imagery vividness interacted with the shape match effect in the early window only: stronger imagers showed a pronounced right-posterior negativity, whereas verbal thinking tendency did not, indicating specificity to visual imagery. Together, results suggest that comparable behavioral outcomes may arise through partly different paths to meaning: an early, visually specific route recruited more strongly by vivid imagers, and a later conceptual route shared more broadly across the imagery spectrum. Language comprehension may draw on both perceptual simulation and linguistic associations, with the relative contribution of each depending in part on the comprehender.

  • Open Access

    The influence of co-speech gestures on speech envelope tracking

    by Jacob P. Momsen, Ying Choon Wu, Seana Coulson


  • Open Access

    From dots to faces: individual differences in visual imagery capacity predict the content of Ganzflicker-induced hallucinations

    by Ana Chkhaidze, Reshanne R Reeder, Connor Gag, Anastasia Kiyonaga, Seana Coulson

    A rapidly alternating red and black display known as Ganzflicker induces visual hallucinations that reflect the generative capacity of the visual system. Individuals vary in their degree of visual imagery, ranging from absent to vivid imagery. Recent proposals suggest that differences in the visual system along this imagery spectrum should also influence the complexity of other internally generated visual experiences. Here, we used tools from natural language processing to analyze free-text descriptions of hallucinations from >4000 participants, asking whether people with different imagery phenotypes see different things in their mind’s eye during Ganzflicker-induced hallucinations. Topic modeling of descriptions revealed that strong imagers described complex, naturalistic content, while weak imagers reported simple geometric patterns. Using crowd-sourced sensorimotor norms, we also found that participants with stronger imagery used language with richer perceptual associations. These findings may reflect individual variation in coordination between early visual areas and higher-order regions relevant for the imagery spectrum.

  • Open Access

    Words and Worlds Both: Dynamic Effects of Distributional and Sensorimotor Information in Semantic Processing

    by Harshada Vinaya, Sean Trott, Diane Pecher, René Zeelenberg, Seana Coulson

    An important issue in the semantic memory literature concerns the relative importance of experience-based sensorimotor versus language corpus-based distributional information in conceptual representations. To explore how each contributes to behavioral and neural responses on a conceptual task, EEG and RTs were recorded as healthy young adults viewed terms for concepts (e.g., “APPLE”) followed by properties (e.g., “red”) and pressed a button to indicate whether the property is true or false for the concept. Next, we constructed a series of mixed effects models of response times (RTs) and single-trial electroencephalogram (EEG) responses to the property words. Distributional models predicted data using semantic distance measures (e.g., between “APPLE” and “red”) derived from language corpus-based measures developed by computational linguists. Sensorimotor models predicted data using sensorimotor distance, a measure based on comparisons of each word’s experiential strength on the perceptual and action-effector dimensions from the crowd-sourced Lancaster Sensorimotor Norms. Statistical model comparison was used to determine whether the data was best fit by Distributional, Sensorimotor, or both sorts of information. In keeping with hybrid accounts of semantic memory, we find that both measures of semantic distance explained unique variance for behavioral and neural measures. Modelling EEG across seven successive 100-ms intervals revealed that the predictors’ temporal dynamics varies between true (APPLE – red) and false (APPLE – black) trials, but showed early sensorimotor activation for both. Results show how linguistic context and task demands modulate the recruitment of different information sources, supporting dynamic hybrid accounts of semantic memory.


  • Open Access

    Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics

    by Jacob P. Momsen, Seana Coulson

    While multisensory super-additivity has been demonstrated in the context of visual articulation, it is unclear whether speech and co-speech gestures are similarly subject to super-additive integration. The current study investigates multisensory integration of speech and bodily gestures, testing whether biological motion signatures of co-speech gestures enhance cortical tracking of the speech envelope. We recorded EEG from 20 healthy adults as they watched a series of audiovisual (AV) discourse clips from four conditions: AV congruent clips with co-speech gestures that were naturally aligned with speech, AV incongruent clips in which gestures were not aligned with the speech, audio-only clips in which speech was delivered in isolation, and video-only clips presenting the gesture content with no accompanying speech. As we hypothesize that the kinematics of co-speech gestures are sufficient to drive gestural enhancement of speech tracking, our clips employed minimalistic “point-light” depictions of a speaker's movements: point-light talkers. Using neural decoder models to predict the amplitude of the speech envelope from EEG elicited in all four conditions, we compared speech reconstruction performance between multisensory (AV congruent) and additive models, that is, those representing the summed neural response across the two unisensory conditions. We found significant improvement in decoder scores for models trained on AV congruent trials relative to both audio-only and additive models. Forward models of brain activity indicated signatures of multisensory integration 140–160 msec following changes to the speech envelope. These results provide novel evidence for a multisensory enhancement effect of co-speech gesture kinematics on continuous speech tracking.

  • Open Access

    Visual Imagery Vividness Predicts the Complexity of Induced Hallucinations

    by Ana Chkhaidze, Anastasia Kiyonaga, Seana Coulson, Reshanne R Reeder

    The current study utilizes the Ganzflicker paradigm—a flickering stimulus that induces visual hallucinations—to provide insight into the internally-generated visual experiences that correlate with individual differences in visual imagery. Here, we analyzed rich narrative descriptions of Ganzflicker hallucinations from 4,365 participants using natural language processing, sensorimotor norms, and AI visualizations. We find that overall perceptual richness and visual detail in descriptions increase with imagery vividness. Examining the specific content of these descriptions reveals that vivid imagers report more face and hand-related content than those with weaker imagery. Exploratory AI-generated visualizations of these descriptions provide additional insights, as those with weak imagery report patterns of simple visual features, like colors and geometric forms, while strong imagers’ hallucinations are filled with complex real-world stimuli. These findings suggest imagery differences may lie not in early visual processing but in the integration of basic visual features into complex object- and scene-level representations.

  • Open Access

    Decoding Prosodic Information from Motion Capture Data: The Gravity of Co-Speech Gestures

    by Jacob P. Momsen, Seana Coulson

    In part due to correspondence in time, seeing how a speaking body moves can impact how speech is apprehended. Despite this, little is known about whether and which specific kinematic features of co-speech movements are relevant for their integration with speech. The current study uses machine learning techniques to investigate how co-speech gestures can be quantified to model vocal acoustics within an individual speaker. Specifically, we address whether kinetic descriptions of human movement are relevant for modeling their relationship with speech in time. To test this, we apply experimental manipulations that either highlight or obscure the relationship between co-speech movement kinematics and downward gravitational acceleration. Across two experiments, we provide evidence that quantifying co-speech movement as a function of its anisotropic relation to downward gravitational forces improves how well those co-speech movements can be used to predict prosodic dimensions of speech, as represented by the low-pass envelope. This study supports theoretical perspectives that invoke biomechanics to help explain speech-gesture synchrony and offers motivation for further behavioral or neuroimaging work investigating audiovisual integration and/or biological motion perception in the context of multimodal discourse.


  • Top Cited Papers
    Open Access

    Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects

    by James A. Michaelov, Megan D. Bardolph, Cyma K. Van Petten, Benjamin K. Bergen, Seana Coulson

    Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.


  • Open Access

    Turing Jest: Do Large Language Models have a Sense of Humor?

    by Sean Trott, Drew Ellen Walker, Seana Coulson

    Humor is an essential aspect of human experience, yet surprisingly little is known about how we recognize and understand humorous utterances. Most theories emphasize the role of incongruity detection and resolution, as well as cognitive capacities like Theory of Mind or pragmatic reasoning. In multiple pre-registered experiments, we ask whether the ability to understand verbal humor can emerge from exposure to purely linguistic input. We find that GPT-3, a large language model (LLM) trained on only language data, exhibits above-chance performance in tasks designed to detect, appreciate, and comprehend jokes. Although GPT-3 falls short of human performance, both humans and LLMs misclassify non-jokes with surprising endings as jokes. Further exploratory analyses reveal a relationship between model size and humor comprehension ability. Results suggest first, that LLMs are surprisingly adept at humor comprehension, and second, that language is not all one needs to “get the joke”.