Jr. John P. Santa Maria
Research Subjects
Jr. John P. Santa Maria has a total of 31 co-authors. Here are the top co-authors by number of shared publications:
Bryce Hill
Department of Psychiatry, Beth Israel Deaconess Medical Center, 330 Brookline Avenue, Boston, MA
1 shared publicationCallie Ginapp
Department of Psychiatry, Beth Israel Deaconess Medical Center, 330 Brookline Avenue, Boston, MA
1 shared publicationChad Chamberlin
Department of Pharmacology, Merck Sharp & Dohme Corp. , Boston, Massachusetts, United States
1 shared publicationCharles Garlisi
Department of Pharmacology, Merck Sharp & Dohme Corp. , Kenilworth, New Jersey, United States
1 shared publicationChristopher L. Smith
Department of Psychiatry, Beth Israel Deaconess Medical Center, 330 Brookline Avenue, Boston, MA
1 shared publicationClifton E. Barry
National Institute of Allergy and Infectious Diseases , Bethesda, Maryland, United States
1 shared publication
Latest Publications
- Open Access
Conversational trajectory degrades large language model detection of suicidal ideation relative to clinicians: a preregistered study
Background: General-purpose large language models increasingly encounter emotional and therapy-like conversation, yet are not developed or evaluated as clinical systems. Existing safety evaluations rely largely on brief exchanges, although harms often unfold over extended interactions. Whether models maintain safety-relevant performance as conversations accumulate context remains unknown. Methods: In this preregistered study, 400 clinician-validated statements, with or without suicidal ideation, were inserted at 0–200 speaker turns in 5 psychotherapy and 3 synthetic transcripts. Forty-nine LLMs and 8 clinicians performed the same binary classification task. Mixed-effects models estimated the effects of conversational depth, model scale, and model version on F1. Twelve top models were tested to 1,500 turns across conversational trajectories, with or without instruction restatement. Results: F1 declined with depth across model families (p<0.001). Larger, newer models performed better but still degraded. Clinicians showed no decline (mean F1 0.86 at both 0 and 200 turns), but eight of nine proprietary models exceeded their performance at 200 turns. Conversational content, not length alone, explained F1 changes; the largest decrease was under adversarial context (p<0.001). Restating instructions increased F1 on therapy to near baseline (median ΔF1 +0.12; p<0.001; 89% median recovery) versus MSJ (ΔF1 +0.08; p=0.04; 38% recovery). Conclusions: LLM detection of suicidal ideation degraded with conversational depth and trajectory, whereas clinician performance remained stable despite the strongest models exceeding most clinicians in absolute performance. Mental health AI safety evaluations should test sustained performance across realistic and adversarial trajectories rather than relying on short-prompt benchmarks. Plain-language summary: Large language models increasingly encounter psychiatric distress and crisis-related content in ordinary use, but existing mental health AI safety benchmarks often rely on isolated prompts, brief exchanges, or simulated conversations rather than real extended interactions. In a preregistered benchmark of 1.4M inferences from 49 LLMs using real psychotherapy transcripts, suicidal-ideation detection declined with conversational depth in every model family while clinician performance remained stable — despite the strongest models matching or exceeding most clinicians in absolute performance — and, among top models, degradation at extended depths depended on the content and direction of the preceding conversation rather than length alone.
Open AccessConversational trajectory degrades large language model detection of suicidal ideation relative to clinicians: a preregistered study
Background: General-purpose large language models increasingly encounter emotional and therapy-like conversation, yet are not developed or evaluated as clinical systems. Existing safety evaluations rely largely on brief exchanges, although harms often unfold over extended interactions. Whether models maintain safety-relevant performance as conversations accumulate context remains unknown. Methods: In this preregistered study, 400 clinician-validated statements, with or without suicidal ideation, were inserted at 0–200 speaker turns in 5 psychotherapy and 3 synthetic transcripts. Forty-nine LLMs and 8 clinicians performed the same binary classification task. Mixed-effects models estimated the effects of conversational depth, model scale, and model version on F1. Twelve top models were tested to 1,500 turns across conversational trajectories, with or without instruction restatement. Results: F1 declined with depth across model families (p<0.001). Larger, newer models performed better but still degraded. Clinicians showed no decline (mean F1 0.86 at both 0 and 200 turns), but eight of nine proprietary models exceeded their performance at 200 turns. Conversational content, not length alone, explained F1 changes; the largest decrease was under adversarial context (p<0.001). Restating instructions increased F1 on therapy to near baseline (median ΔF1 +0.12; p<0.001; 89% median recovery) versus MSJ (ΔF1 +0.08; p=0.04; 38% recovery). Conclusions: LLM detection of suicidal ideation degraded with conversational depth and trajectory, whereas clinician performance remained stable despite the strongest models exceeding most clinicians in absolute performance. Mental health AI safety evaluations should test sustained performance across realistic and adversarial trajectories rather than relying on short-prompt benchmarks. Plain-language summary: Large language models increasingly encounter psychiatric distress and crisis-related content in ordinary use, but existing mental health AI safety benchmarks often rely on isolated prompts, brief exchanges, or simulated conversations rather than real extended interactions. In a preregistered benchmark of 1.4M inferences from 49 LLMs using real psychotherapy transcripts, suicidal-ideation detection declined with conversational depth in every model family while clinician performance remained stable — despite the strongest models matching or exceeding most clinicians in absolute performance — and, among top models, degradation at extended depths depended on the content and direction of the preceding conversation rather than length alone.
- 14 August 2017
- Published by American Chemical Society (ACS) in ACS Chemical Biology
- Vol. 12 (9) , 2448-2456
Linking High-Throughput Screens to Identify MoAs and Novel Inhibitors of Mycobacterium tuberculosis Dihydrofolate Reductase
Though phenotypic and target-based high-throughput screening approaches have been employed to discover new antibiotics, the identification of promising therapeutic candidates remains challenging. Each approach provides different information, and understanding their results can provide hypotheses for a mechanism of action (MoA) and reveal actionable chemical matter. Here, we describe a framework for identifying efficacy targets of bioactive compounds. High throughput biophysical profiling against a broad range of targets coupled with machine learning was employed to identify chemical features with predicted efficacy targets for a given phenotypic screen. We validate the approach on data from a set of 55 000 compounds in 24 historical internal antibacterial phenotypic screens and 636 bacterial targets screened in high-throughput biophysical binding assays. Models were built to reveal the relationships between phenotype, target, and chemotype, which recapitulated mechanisms for known antibacterials. We also prospectively identified novel inhibitors of dihydrofolate reductase with nanomolar antibacterial efficacy against Mycobacterium tuberculosis. Molecular modeling provided structural insight into target–ligand interactions underlying selective killing activity toward mycobacteria over human cells.
- 14 August 2017
- Published by American Chemical Society (ACS) in ACS Chemical Biology
- Vol. 12 (9) , 2448-2456
Linking High-Throughput Screens to Identify MoAs and Novel Inhibitors of Mycobacterium tuberculosis Dihydrofolate Reductase
Though phenotypic and target-based high-throughput screening approaches have been employed to discover new antibiotics, the identification of promising therapeutic candidates remains challenging. Each approach provides different information, and understanding their results can provide hypotheses for a mechanism of action (MoA) and reveal actionable chemical matter. Here, we describe a framework for identifying efficacy targets of bioactive compounds. High throughput biophysical profiling against a broad range of targets coupled with machine learning was employed to identify chemical features with predicted efficacy targets for a given phenotypic screen. We validate the approach on data from a set of 55 000 compounds in 24 historical internal antibacterial phenotypic screens and 636 bacterial targets screened in high-throughput biophysical binding assays. Models were built to reveal the relationships between phenotype, target, and chemotype, which recapitulated mechanisms for known antibacterials. We also prospectively identified novel inhibitors of dihydrofolate reductase with nanomolar antibacterial efficacy against Mycobacterium tuberculosis. Molecular modeling provided structural insight into target–ligand interactions underlying selective killing activity toward mycobacteria over human cells.