EPEllie Pavlick

Ellie Pavlick

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Ellie Pavlick has a total of 107 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Neuro-Symbolic Knowledge Graphs Improve Motivational Interviewing Fidelity in Clinical Artificial Intelligence Tools: Development and Simulation Study (Preprint)

    by Jun Tao, Trisha Arnold, Hao Wu, Hannah Parent, Mattia Prosperi, Natalie Fenn, Tyler Wray, Ellie Pavlick, Bryce K. Perler, Jack Rusley, et al.

    BACKGROUND Large language models (LLMs) are increasingly used in health counseling, but adherence to evidence-based approaches such as motivational interviewing (MI) is inconsistent, and their reasoning is difficult to inspect or audit. Supervised fine-tuning (SFT) trains a model to imitate expert responses but does not explicitly encode the sequential clinical decisions MI requires. Knowledge graphs (KGs), which represent clinical knowledge as explicit, inspectable rules and relationships, may offer a more transparent way to govern LLM counseling. OBJECTIVE This study aimed to develop a neuro-symbolic KG encoding MI clinical logic and to evaluate, in simulated sessions, whether KG guidance improves MI fidelity compared with a base LLM and SFT, alone and in combination. METHODS We built a 3-layer KG comprising an MI ontology (34 techniques, stages of change, and patient talk types), a deterministic rule engine (34 decision rules governing technique selection and phase progression), and 13 hard safety constraints, and integrated it into CHIA (Chatbot for HIV Prevention and Action), a preexposure prophylaxis (PrEP) counseling platform. Using a 2×2 factorial design crossing SFT (GPT-4.1 base vs fine-tuned on 2400 expert MI examples) with KG guidance (disabled vs enabled), we simulated 8-turn sessions with 20 LLM-generated patient personas (3 runs per persona per arm; 239/240 sessions completed). A blinded LLM judge (Claude Sonnet 4.5) scored Motivational Interviewing Treatment Integrity (MITI) 4.2 global scores (Empathy, Partnership, Cultivating Change Talk, and Softening Sustain Talk) and flagged premature planning. We used 2-way repeated-measures ANOVA and generalized estimating equations with false discovery rate correction, and computed linguistic metrics including lexical diversity. A blinded MI expert and a second LLM judge (Gemini 2.5 Pro) rescored a stratified subsample of 40 sessions. RESULTS The base LLM with KG guidance (Base+KG) achieved the highest mean Empathy (4.90, SD 0.30) and Partnership (4.88, SD 0.32). Compared with SFT alone, Base+KG showed higher Empathy (Cohen d=0.68, 95% CI 0.37-1.01) and Partnership (d=0.91, 95% CI 0.60-1.25). SFT had significant negative main effects on Empathy (F1,19=21.26; P<.001) and Partnership (F1,19=17.39; P<.001) and reduced lexical diversity (d=−1.47). The LLM judge flagged premature planning in 0/60 Base+KG sessions vs 1/60 Base, 4/59 SFT, and 6/60 SFT+KG sessions; however, the human expert identified more premature planning in every arm. Human-LLM agreement was moderate for Empathy (intraclass correlation coefficient [ICC] 0.55) and Partnership (ICC 0.42) and poor for Softening Sustain Talk, but all 3 raters produced consistent arm rankings. CONCLUSIONS In this simulation study, KG guidance applied to a base LLM yielded higher MI fidelity than SFT, which degraded relational quality. Externalizing clinical reasoning into an inspectable KG is a promising, auditable approach to governing LLM counseling, pending confirmation with human participants and larger human-rated samples.

  • Open Access

    From lectures to learning outcomes: meaningful integration of AI-generated content in pre-clerkship medical training

    by Jay Khurana, Hossam A Zaki, Ellie Pavlick, Jillian Turbitt, Heather McGee, Sahil Gupta, Sriya Sai Pushpa Datla, Salma Eldeeb, Thais Salazar Mather, Sarita Warrier, et al.


  • Top Cited Papers
    Open Access

    Evaluation of an Artificial Intelligence Conversational Chatbot to Enhance HIV Preexposure Prophylaxis Uptake: Development and Usability Internal Testing

    by Jun Tao, Ellie Pavlick, Amaris Grondin, Josue D Bustamante, Harrison Martin, Hannah Parent, Natalie Fenn, Alexi Almonte, Amanda Maguire-Wilkerson, Mofan Gu, et al.

    Background: The HIV epidemic in the United States disproportionately impacts gay, bisexual, and other men who have sex with men (MSM). Despite the effectiveness of HIV preexposure prophylaxis (PrEP) in preventing HIV acquisition, uptake among MSM remains suboptimal. Motivational interviewing (MI) has demonstrated efficacy at increasing PrEP uptake among MSM but is resource-intensive, limiting scalability. The use of artificial intelligence, particularly large language models with conversational agents (ie, “chatbots”) such as ChatGPT, may offer a scalable approach to delivering MI-based counseling for PrEP and HIV prevention. Objective: This internal usability testing aimed to evaluate the development of an artificial intelligence–based chatbot, including its ability to provide MI-aligned education about PrEP and HIV prevention and potential to support PrEP uptake. Methods: The Chatbot for HIV Prevention and Action (CHIA) was built on a GPT-4o base model embedded with a validated knowledge database on HIV and PrEP in English and Spanish. The CHIA was fine-tuned through training on a large MI dataset and prompt engineering. The use of the AutoGen multiagent framework enabled the CHIA to integrate 2 agents, the PrEP Counselor Agent and the Assistant Agent, which specialized in providing MI-based counseling and handling function calls (eg, assessment of HIV risk), respectively. During internal testing from March 10-April 28, 2025, we systematically evaluated the CHIA’s performance in English and Spanish using a set of 5-point Likert scales to measure accuracy, conciseness, up-to-dateness, trustworthiness, and alignment with aspects of the MI spirit (eg, collaboration, autonomy support) and MI-consistent behaviors (eg, affirmation, open-ended questions). Descriptive statistics and mixed linear regression were used to analyze the data. Results: A total of 296 responses, including 145 English responses and 151 Spanish responses, were collected during the internal testing period. Overall, the CHIA demonstrated strong performance across both languages, receiving the highest combined scores in the general response quality metrics including up-to-dateness (mean 4.6, SD 0.8), trustworthiness (mean 4.5, SD 0.9), accuracy (mean 4.4, SD 0.9), and conciseness (mean 4.2, SD 1.1). The CHIA generally received higher combined scores for metrics that assessed alignment with the MI spirit (ie, empathy, evocation, autonomy support, and collaboration) and lower combined scores for MI-consistent behaviors (ie, affirmation, open-ended questions, and reflections). Spanish responses had significantly lower mean scores than English responses across nearly all MI-based metrics. Conclusions: Our internal usability testing highlights the potential of the CHIA as a viable tool for delivering MI-aligned counseling in English and Spanish to promote HIV prevention and support PrEP uptake, though its Spanish language performance requires further improvement.


  • Open Access

    Video Finetuning Improves Reasoning Between Frames

    by Ruiqi Yang, Tian Yun, Zihan Wang, Ellie Pavlick

    Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame tokens. In this work, we investigate what video finetuning brings to multimodal LLMs. We propose Visual Chain-of-Thought (vCoT), an explicit reasoning process that generates transitional event descriptions between consecutive frames. Using vCoT, we systematically compare image-only LVLMs with their video-finetuned counterparts, both with and without access to these transitional cues. Our experiments show that vCoT significantly improves the performance of image-only models on long-form video question answering, while yielding only marginal gains for video-finetuned models. This suggests that the latter already capture frame-to-frame transitions implicitly. Moreover, we find that video models transfer this temporal reasoning ability to purely static settings, outperforming image models' baselines on relational visual reasoning tasks.

  • Open Access

    Parallel trade-offs in human cognition and neural networks: The dynamic interplay between in-context and in-weight learning

    by Jacob Russin, Ellie Pavlick, Michael J. Frank

    Human learning embodies a striking duality: Sometimes, we can rapidly infer and compose logical rules, benefiting from structured curricula (e.g., in formal education), while other times, we rely on an incremental approach or trial-and-error, learning better from curricula that are randomly interleaved. Influential psychological theories explain this seemingly conflicting behavioral evidence by positing two qualitatively different learning systems—one for rapid, rule-based inferences (e.g., in working memory) and another for slow, incremental adaptation (e.g., in long-term and procedural memory). It remains unclear how to reconcile such theories with neural networks, which learn via incremental weight updates and are thus a natural model for the latter, but are not obviously compatible with the former. However, recent evidence suggests that metalearning neural networks and large language models are capable of in-context learning (ICL)—the ability to flexibly infer the structure of a new task from a few examples. In contrast to standard in-weight learning (IWL), which is analogous to synaptic change, ICL is more naturally linked to activation-based dynamics thought to underlie working memory in humans. Here, we show that the interplay between ICL and IWL naturally ties together a broad range of learning phenomena observed in humans, including curriculum effects on category-learning tasks, compositionality, and a trade-off between flexibility and retention in brain and behavior. Our work shows how emergent ICL can equip neural networks with fundamentally different learning properties that can coexist with their native IWL, thus offering an integrative perspective on dual-process theories of human cognition.


  • Open Access

    What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Models

    by Tian Yun, Chen Sun, Ellie Pavlick

    Recent work has argued that large language models (LLMs) are not "abstract reasoners", citing their poor zero-shot performance on a variety of challenging tasks as evidence. We revisit these experiments in order to add nuance to the claim. First, we show that while LLMs indeed perform poorly in a zero-shot setting, even tuning a small subset of parameters for input encoding can enable near-perfect performance. However, we also show that this finetuning does not necessarily transfer across datasets. We take this collection of empirical results as an invitation to (re-)open the discussion of what it means to be an "abstract reasoner", and why it matters whether LLMs fit the bill.

  • Open Access

    LLMs model how humans induce logically structured rules

    by Alyssa Loo, Ellie Pavlick, Roman Feiman

    A central goal of cognitive science is to provide a computationally explicit account of both the structure of the mind and its development: what are the primitive representational building blocks of cognition, what are the rules via which those primitives combine, and where do these primitives and rules come from in the first place? A long-standing debate concerns the adequacy of artificial neural networks as computational models that can answer these questions, in particular in domains related to abstract cognitive function, such as language and logic. This paper argues that recent advances in neural networks -- specifically, the advent of large language models (LLMs) -- represent an important shift in this debate. We test a variety of LLMs on an existing experimental paradigm used for studying the induction of rules formulated over logical concepts. Across four experiments, we find converging empirical evidence that LLMs provide at least as good a fit to human behavior as models that implement a Bayesian probablistic language of thought (pLoT), which have been the best computational models of human behavior on the same task. Moreover, we show that the LLMs make qualitatively different predictions about the nature of the rules that are inferred and deployed in order to complete the task, indicating that the LLM is unlikely to be a mere implementation of the pLoT solution. Based on these results, we argue that LLMs may instantiate a novel theoretical account of the primitive representations and computations necessary to explain human logical concepts, with which future work in cognitive science should engage.

  • Open Access

    How Do Vision-Language Models Process Conflicting Information Across Modalities?

    by Tianze Hua, Tian Yun, Ellie Pavlick

    AI models are increasingly required to be multimodal, integrating disparate input streams into a coherent state representation on which subsequent behaviors and actions can be based. This paper seeks to understand how such models behave when input streams present conflicting information. Focusing specifically on vision-language models, we provide inconsistent inputs (e.g., an image of a dog paired with the caption "A photo of a cat") and ask the model to report the information present in one of the specific modalities (e.g., "What does the caption say / What is in the image?"). We find that models often favor one modality over the other, e.g., reporting the image regardless of what the caption says, but that different models differ in which modality they favor. We find evidence that the behaviorally preferred modality is evident in the internal representational structure of the model, and that specific attention heads can restructure the representations to favor one modality over the other. Moreover, we find modality-agnostic "router heads" which appear to promote answers about the modality requested in the instruction, and which can be manipulated or transferred in order to improve performance across datasets and modalities. Together, the work provides essential steps towards identifying and controlling if and how models detect and resolve conflicting signals within complex multimodal environments.

  • Open Access

    Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline

    by Meng Lu, Ruochen Zhang, Carsten Eickhoff, Ellie Pavlick

    Multilingual large language models (LLMs) often exhibit factual inconsistencies across languages, with significantly better performance in factual recall tasks in English than in other languages. The causes of these failures, however, remain poorly understood. Using mechanistic analysis techniques, we uncover the underlying pipeline that LLMs employ, which involves using the English-centric factual recall mechanism to process multilingual queries and then translating English answers back into the target language. We identify two primary sources of error: insufficient engagement of the reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer. To address these vulnerabilities, we introduce two vector interventions, both independent of languages and datasets, to redirect the model toward better internal paths for higher factual consistency. Our interventions combined increase the recall accuracy by over 35 percent for the lowest-performing language. Our findings demonstrate how mechanistic insights can be used to unlock latent multilingual capabilities in LLMs.

  • Open Access

    From Lectures to Learning Outcomes: Meaningful Integration of AI-Generated Content in Pre-Clerkship Medical Training

    by Jay Khurana, Hossam A Zaki, Ellie Pavlick, Jillian Turbitt, Heather McGee, Sahil Gupta, Sriya Sai Pushpa Datla, Salma Eldeeb, Thais Salazar Mather, Sarita Warrier, et al.

    Large Language Models (LLMs) have shown considerable promise in knowledge processing and synthesis across various medical disciplines. In medical education, most applications have focused on comparing LLM outputs to trainee performance or using LLMs for standardized assessment. However, few studies have systematically evaluated the effects of standardized, LLM-powered, curricular interventions on medical learning. This case study, conducted at The Warren Alpert Medical School of Brown University, assessed the impact of AI-generated Anki flashcards and lecture summaries specifically optimized for the pre-clerkship phase. These materials were developed using a rigorous, specific, and content-agnostic prompt engineering process and validated through standardized human grading to ensure both accuracy and relevance. The final prompts used demonstrated hallucination rates of 0 per summary and 1 per 21 flashcards and average coverage of 100% of faculty-identified learning objectives. Materials were given to students for two 3-week academic blocks, covering genetics and pharmacology. Student exam scores and survey-based feedback were used to evaluate the effectiveness of these AI-generated resources. The study was conducted in a resource-rich pre-clerkship setting where students already have access to faculty-created materials, commercial content, and student-curated resources. We aimed to determine whether AI-generated content could offer measurable quantitative improvements or subjective qualitative benefits in a saturated learning environment. Among participating first-year medical students, overall exam performance between those who used the AI-generated summaries and those who did not was comparable in both the genetics block (p = 0.76) and the pharmacology block (p = 0.35). Similarly, use of the AI-generated Anki flashcards was not associated with significant differences in exam scores for either genetics (p = 0.86) or pharmacology (p = 0.05). Qualitative analyses demonstrated widespread time saving for Anki flashcards (74%) and AI-generated summaries (61%), with 91% of users finding the custom AI-generated content more time-saving than default GPT-4o. There was a significant usage-dependent relationship of higher AI-usage correlating with increased agreement of equivalency or utility over faculty-generated lecture notes (Pearson’s r 2 =0.55) and student-created flashcards (Pearson’s r 2 =0.79). These findings suggest that students who used AI-generated content maintained comparable educational outcomes in the pre-clerkship setting. Moreover, subjective perceptions among learners, such as time saved and content usefulness, highlight the potential value of LLM-powered tools when layered on top of an existing well-resourced curricular structure. Future work will examine the benefits of this work in less structured medical education settings, such as clinical and surgical education.