H. Andrew Schwartz

Humans use language to communicate psychological experiences. Advancements in Natural LanguageProcessing have enabled researchers to quantitatively study and assess psychological states andbehaviors through language (e.g., well-being and mental health). Insights from language analyses areoften best conveyed through visualizations. Data-driven visualizations of how language statisticallyrelates to psychological dimensions can help interpret assessment scores, uncover meaningful patterns,and understand the intricate relationships between language use and mental states. Visualizations canbe used to differentiate between related constructs (e.g., How does language related to depression versusanxiety differ?), understand the validity of assessment tools (e.g., Does the language related to highdepression severity match with depression theory?), and explain language-based assessments (e.g.,Which language is indicative of higher depression severity in a language-based assessment?).This tutorial demonstrates how to transform language data into visual insights using the text and topicspackages in R. We provide practical guidance on creating four types of visualizations based on differentlanguage analysis techniques, examining how language use patterns are statistically associated with apsychological construct. We visualize 1) individual words and phrases (n-grams), 2) topics (wordclusters), 3) words in the word embedding space, and 4) relevant language examples. These differentvisualizations offer complementary perspectives, visualizing linguistic elements from individual wordsto overarching topics and text examples. The underlying methods range from correlations betweenindividual words to more advanced techniques, including topic modeling and Large Language Models.By making these methods accessible, this tutorial aims to empower researchers to use languagevisualizations for meaningful, data-driven insights.

Self-reported rating scales have been central to social science for decades, even though language is our primary form of communication. We used language analysis with machine learning to compare self-reported ratings with language-based responses of experienced well-being (i.e., daily emotions) linked with human traits, states, and behaviors. In a sample of 764 U.S. service workers who reported their feelings daily for up to 12 weeks, responses were provided in 1) numerical ratings, 2) short essays, and 3) descriptive words. When predicting nine self-rated affective traits, essays were slightly less accurate than the ratings (r = .49 v .55, p < .001), while the word-based responses slightly outperformed the ratings (r = .53 vs .51, p < .001). For external predictive validity across 24 traits, demographics, and behaviors, both language formats doubled the ratings’ accuracy, with words reaching an average of r = .18 and essays r = .24 (p < .001). Beyond their greater accuracy, both language formats offered rich psychological insights, with words focusing on emotions and essays encompassing broader life domains. While both language-based formats showed advantages and disadvantages, each demonstrated greater information-richness than traditional rating scales.

Background: Machine learning and natural language processing have demonstrated significant potential for mental health assessment: describing your mental health in your own words can offer a more ecologically valid approach than traditional rating scales. However, most models focus on specific diagnoses, conditions, or symptoms, which may prematurely assign labels and potentially reinforce stigma in the context of early-stage mental health screening. Objective: This study develops a language-based assessment model that assesses the need for mental health support based on probed natural language and validates it against best-estimate assessments from multiple experienced psychotherapists. Methods: We analyzed an enriched online sample (n=600 for development and n=212 for validation), in which about half reported experiencing internalizing symptoms (depression or anxiety). Participants described their mental health using open-ended responses regarding (1) mental health, (2) suicidal thoughts, (3) medical history, and (4) depression. The responses were converted into contextual word embeddings using a large language model and entered as predictors in a ridge regression using nested cross-validation. Two to three experienced psychotherapists assessed each participant’s need for mental health support on a scale from 1 (no support needed) to 5 (potential crisis). Their assessments were based on longitudinal clinical data (natural language, validated scales, clinical interview, sociodemographics, and clinical history) and were averaged into a best-estimate assessment for model validation. We used the Sequential Evaluation With Model Preregistration framework, which separates model development from validation in a held-out set to support robust estimations and generalizability. Results: The language-based assessments closely aligned with the best-estimate assessments ( r =0.82) and showed strong correlations with established clinical rating scales for depression (Patient Health Questionnaire-9), anxiety (Generalized Anxiety Disorder 7-Item Scale), stress (Perceived Stress Scale 10), and suicidality (Inventory of Depression and Anxiety Symptoms; r =0.62-0.77). Language-based visualizations of topics and word embeddings showed that low need for support assessments was associated with mentioning well-being and good health, while high assessments were related to depression, anxiety, and suicidality. Conclusions: This study demonstrates that natural language responses analyzed through large language models and machine learning can be used to assess individuals’ need for mental health support in close alignment with best-estimate assessments from experienced psychotherapists. Using less than 5 minutes of respondent time, this approach offers a practical tool for early-stage mental health screening in both clinical and self-guided screening contexts.

Language-based assessments (LBAs) have shown strong convergent and criterion validity cross-sectionally and longitudinally. However, no study has tested the ability of LBAs of affect to capture experiment-induced changes (i.e., causal validity). In this randomised mixed-design experiment, participants (N=892) completed a Mood Induction Procedure comprising different conditions (church, shopping mall, and park) using physical locations (N=153) and online video clips (N=739). Participants reported their pre- and post-induction affect through open-ended responses and ratings (i.e., the Positive and Negative Affect Schedule; PANAS). Using large language models and machine learning, we compared how accurately PANAS and language responses classified the experimental conditions. The LBA derived from the language responses exhibited significantly higher accuracy compared to the PANAS scores in classifying the conditions in both the training set (cross-validated AUC=.74 vs. .63), the online holdout set (AUC = .76 vs. .70), and the offline holdout set (AUC = .67 vs. .53). Apart from showing higher predictive accuracy, we assessed LBAs of specific affective dimensions (valence, arousal, and basic emotions). These analyses revealed that positive valence was most strongly expressed in the park condition, while anger was particularly elevated in the mall condition. Further, we visualized the language differences across conditions, showing more calm and peaceful expressions in the park condition and more tired and bored expressions in the mall condition. These findings support the causal validity of language-based affect assessments and its ability to provide insights into affective states. We discuss the potential of using LBA in experimental psychological research.

Some psychotherapies, such as written exposure therapy for posttraumatic stress disorder, utilize “scripts” during parts of treatment, but verifying script adherence to ensure engagement of key mechanisms of change is a time consuming step for therapy supervisors. Here, we formalize therapy script adherence as an NLP task, and evaluate several simple (text similarity) and more complex (few-shot LLM) approaches. Over 351 annotated therapist utterance-script pairs, we find text similarity approaches to be highly competitive with LLMs and produce fewer false positives. ROUGE-L recall achieves F1 = 0.973, and BLEU achieves F1 = 0.972 with full precision and zero false positives. GPT-5.2 achieves F1 = 0.935 and GPT-4o-mini achieves F1 = 0.876. Given that the text similarity techniques are multiple orders of magnitude less complex, our results underscore the ability of simpler NLP techniques to still be effective in the age of LLMs for tasks that are more textual in nature and suggest their utility in clinical training for giving supervisees rapid, privacy-preserving feedback on their delivery of evidence-based psychotherapy scripts.

Intolerance of uncertainty (IU) is central to cognitive models of posttraumatic stress disorder (PTSD), reflecting a maladaptive process that maintains hypervigilant threat appraisals and symptom persistence. Traditional IU self-report assessments, however, are constrained to explicit, trait-like evaluations and may not capture IU’s implicit and context-dependent nature. Ecological momentary assessment (EMA) combined with large language models (LLMs) offers a scalable approach for detecting subtle linguistic markers of IU in daily life. Here, we introduce a novel language-based method for monitoring IU through naturalistic daily diaries and examine its role in perpetuating PTSD symptoms. In a 90-day EMA study of 432 trauma-exposed individuals with elevated PTSD symptoms (29,623 diary entries with 32,941 daily PTSD symptom ratings), we developed an LLM-based IU assessment, yielding the LLM-based similarity between daily transcribed speech and established IU scale language (r = .60 between LLM measure and human ratings). Using dynamic structural equation modeling (DSEM), we found that higher daily IU predicted greater next-day PTSD symptom severity (β = 0.041, 95% CI [0.024, 0.056]), even after controlling for prior-day symptoms. Conversely, higher daily PTSD symptom severity predicted greater next-day IU (β = 0.035, 95% CI [0.018, 0.054]). The strength of these bidirectional effects varied across individuals (IU→PTSD variance = 0.014, 95% CI [0.007, 0.022]; PTSD→IU variance = 0.037, 95% CI [0.030, 0.047]), reflecting substantial heterogeneity in daily IU–PTSD dynamics. Finally, individuals with stronger IU persistence (IUₜ₋₁ → IUₜ) tended to exhibit enhanced same-day IU–PTSD associations (IUₜ ↔ PCLₜ; r = 0.23, 95% CI = [0.016, 0.410]). In contrast, greater PTSD persistence (PCLₜ₋₁ → PCLₜ) covaried negatively with both lagged (IUₜ₋₁ → PCLₜ; r = -0.38, 95% CI = [-0.505, -0.246]) and contemporaneous (IUₜ ↔ PCLₜ; r = -0.18, 95% CI = [-0.327, -0.024]) IU–PTSD effects, suggesting divergent dynamic patterns relative to IU. Complementary multilevel analyses demonstrated that language-derived IU provides incremental validity, explaining variance in PTSD symptom severity beyond sentiment (NRCSent) and positive and negative affect (LIWC). These findings extend cognitive models of PTSD with IU as a dynamic mechanism influencing symptom trajectories in daily life and demonstrate an ecologically valid framework for tracking implicit cognitive processes via natural language.

Artificial intelligence (AI) systems and large language models (LLMs) offer substantial potential to augment or even fundamentally change elements of psychological assessment and treatment. However, current AI technologies have yet to demonstrate the capacity to effect meaningful and sustained clinical change. This gap reflects both the limited integration of clinical science knowledge into language models and applications built using them, as well as the mismatch between the brief, minutes-long nature of most AI interactions and the months-long course of most evidence-based treatments. Here we introduce the Evidence-Based Psychotherapy with AI (EBP-AI) framework, which articulates a set of principles for developing effective clinical AI applications: 1) psychodiagnostic assessment, 2) longitudinal case conceptualization, 3) appropriately dosed intervention planning, 4) meaningful progress evaluation, 5) rigorous validation with clinical populations, 6) attention to real world implementation and use, 7) clinically appropriate style, and 8) understanding clinical psychology as a living science. We introduce a set of key technical questions for the development and evaluation of clinical LLMs and AIs aligned with these principles. Despite their potential, current clinical AIs fall short, in part due to issues with memory, sycophancy, and prioritizing short-term helpfulness over long-term clinical impact. Responsible and ethical design of effective, clinical-science-based AI systems will require understanding their limitations and strategically extending their capabilities.

Measuring the subjective well-being of societies is important in its own right and as a determinant of health outcomes. Traditionally, self-report surveys such as Gallup’s daily poll have tracked well-being, but they are expensive and provide limited coverage of communities. Since 2010, research has shown that social media (e.g., Twitter/X) offers a cost-effective alternative. Yet, through ownership and policy changes, access to these “digital commons” is increasingly restricted. What are we losing without access for researchers? We document that well-being estimates derived from archival geolocated Twitter language likely yielded the most valid indicators of US population well-being available. We compare demographically post-stratified estimates for 1,208 US counties (~89% of the population) derived from 1.53 billion posts by 5.25 million users from 2009-2015 with Gallup estimates based on 1.9 million survey responses (2009-2016). Twitter-based estimates were more predictive of external health and economic indicators and met a wide variety of validity criteria, including a convergent correlation of r = .70 with Gallup, high test-retest stability, linguistic face validity, and generalizability across US cities, states, and to the UK. We further show that, even without Gallup data, Twitter/X data sufficed to measure well-being using an independent language model trained on N = 9,419 Twitter users, which again outperformed Gallup in predicting external variables. These findings establish that social media can capture population well-being more robustly and with better coverage than even the largest survey efforts, and that, like other public health datasets, researcher access to these data should be protected.

Background. Mental and physical health are correlated, yet are described differently in natural language. Standard symptom checklists may limit individuals from fully describing how they experience health problems. This study examines the distinction between how individuals linguistically describe mental and physical health.Method. We used the Sequential Evaluation with Model Pre-registration (SEMP) design to analyze language from automated, open-ended clinical interviews with 9/11 first responders. We developed language-based models assessing Mental and Physical Component Summary (M/PCS) scores from automated interview transcripts (N=1290). We then evaluated these models on held-out prospective data (N = 310) against M/PCS, medical record diagnoses, and healthcare expenditures.Results. Language-based assessment correlated significantly with M/PCS scores (r = .36 and r = .37, respectively, all p<.001), exceeding SEMP pre-registered thresholds (r>.315, and r>.348, respectively). Analyses revealed both shared and distinct linguistic markers for mental and physical health, with language reflecting emotional, somatic, and functional themes. The language-based assessments demonstrated external validity through significant convergence with clinical diagnoses (AUCs=.74 and .65 for mental and physical health, respectively). Furthermore, language-based assessments exhibited incremental validity; when added to models containing traditional self-reports, they explained significant unique variance in both mental (ΔR²=.063, p<.001) and physical (ΔR²=.032, p=.010) healthcare costs.Conclusions. Natural language from automated interviews captures general health through emotional, somatic, and functional themes. Validated against clinical diagnoses and healthcare costs, these models demonstrate that open-ended language provides interpretable insights beyond traditional rating scales, potentially augmenting clinical evaluations and understanding.

Introduction: Clinical trial endpoints are a ‘finite sequence of instructions to perform a task’ (measure treatment effectiveness), making them algorithms. Consequently, they may exhibit algorithmic bias: internal and external performance can vary across demographic groups, impacting fairness, validity and clinical decision‐making. Methods: We developed the open‐source Detecting Algorithmic Bias (DAB) Pipeline in Python to identify endpoint ‘performance variance’—a specific algorithmic bias—as the proportion of minority participants changes. This pipeline assesses internal performance (on demographically matched test data) and external performance (on demographically diverse validation data) using metrics including F1 scores and area under the receiver operating characteristic curve (AUROC). We applied it to representative opioid use disorder (OUD) trial endpoints. Results: F1 scores remained stable across minority representation levels, suggesting consistency in precision‐recall balance (F1) despite demographic shifts. Conversely, AUROC measures were more sensitive, revealing significant performance variance. Training on demographically homogeneous populations boosted internal performance (accuracy within similar cohorts) but critically compromised external generalisability (accuracy within diverse cohorts). This pattern reveals an ‘endpoint bias trade‐off’: optimising performance for homogeneous populations vs. having generalisable performance for the real world. Discussion and Conclusions: Consistently performing endpoints for one demographic profile may lose generalisability during population shifts, potentially introducing endpoint bias. Increasing minority representation in the training data consistently improved generalisability. The endpoint bias trade‐off reinforces the importance of diverse recruitment in OUD trials. The DAB Pipeline helps researchers systematically pinpoint when an endpoint may suffer ‘performance variance’ (i.e., bias). As an open‐source tool, it promotes transparent endpoint evaluation and supports selecting demographically invariant OUD endpoints.