CLChristine Lee

Christine Lee

*This is an algorithmically generated author record based on publicly available information.

Christine Lee has a total of 16 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Large Language Models Versus Expert Clinicians in Crisis Prediction Among Telemental Health Patients: Comparative Study

    by Christine Lee, Matthew Mohebbi, Erin O'CAllaghan, Mirène Winsberg

    Background: Due to recent advances in artificial intelligence, large language models (LLMs) have emerged as a powerful tool for a variety of language-related tasks, including sentiment analysis, and summarization of provider-patient interactions. However, there is limited research on these models in the area of crisis prediction. Objective: This study aimed to evaluate the performance of LLMs, specifically OpenAI’s generative pretrained transformer 4 (GPT-4), in predicting current and future mental health crisis episodes using patient-provided information at intake among users of a national telemental health platform. Methods: Deidentified patient-provided data were pulled from specific intake questions of the Brightside telehealth platform, including the chief complaint, for 140 patients who indicated suicidal ideation (SI), and another 120 patients who later indicated SI with a plan during the course of treatment. Similar data were pulled for 200 randomly selected patients, treated during the same time period, who never endorsed SI. In total, 6 senior Brightside clinicians (3 psychologists and 3 psychiatrists) were shown patients’ self-reported chief complaint and self-reported suicide attempt history but were blinded to the future course of treatment and other reported symptoms, including SI. They were asked a simple yes or no question regarding their prediction of endorsement of SI with plan, along with their confidence level about the prediction. GPT-4 was provided with similar information and asked to answer the same questions, enabling us to directly compare the performance of artificial intelligence and clinicians. Results: Overall, the clinicians’ average precision (0.7) was higher than that of GPT-4 (0.6) in identifying the SI with plan at intake (n=140) versus no SI (n=200) when using the chief complaint alone, while sensitivity was higher for the GPT-4 (0.62) than the clinicians’ average (0.53). The addition of suicide attempt history increased the clinicians’ average sensitivity (0.59) and precision (0.77) while increasing the GPT-4 sensitivity (0.59) but decreasing the GPT-4 precision (0.54). Performance decreased comparatively when predicting future SI with plan (n=120) versus no SI (n=200) with a chief complaint only for the clinicians (average sensitivity=0.4; average precision=0.59) and the GPT-4 (sensitivity=0.46; precision=0.48). The addition of suicide attempt history increased performance comparatively for the clinicians (average sensitivity=0.46; average precision=0.69) and the GPT-4 (sensitivity=0.74; precision=0.48). Conclusions: GPT-4, with a simple prompt design, produced results on some metrics that approached those of a trained clinician. Additional work must be done before such a model can be piloted in a clinical setting. The model should undergo safety checks for bias, given evidence that LLMs can perpetuate the biases of the underlying data on which they are trained. We believe that LLMs hold promise for augmenting the identification of higher-risk patients at intake and potentially delivering more timely care to patients.


  • Open Access

    Large Language Models Versus Expert Clinicians in Crisis Prediction Among Telemental Health Patients: Comparative Study (Preprint)

    by Christine Lee, Matthew Mohebbi, Erin O'Callaghan, Mirène Winsberg

    Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Due to recent advances in artificial intelligence, large language models (LLMs) have emerged as a powerful tool for a variety of language-related tasks, including sentiment analysis, and summarization of provider-patient interactions. However, there is limited research on these models in the area of crisis prediction. Objective: This study aimed to evaluate the performance of LLMs, specifically OpenAI’s generative pretrained transformer 4 (GPT-4), in predicting current and future mental health crisis episodes using patient-provided information at intake among users of a national telemental health platform. Methods: Deidentified patient-provided data were pulled from specific intake questions of the Brightside telehealth platform, including the chief complaint, for 140 patients who indicated suicidal ideation (SI), and another 120 patients who later indicated SI with a plan during the course of treatment. Similar data were pulled for 200 randomly selected patients, treated during the same time period, who never endorsed SI. In total, 6 senior Brightside clinicians (3 psychologists and 3 psychiatrists) were shown patients’ self-reported chief complaint and self-reported suicide attempt history but were blinded to the future course of treatment and other reported symptoms, including SI. They were asked a simple yes or no question regarding their prediction of endorsement of SI with plan, along with their confidence level about the prediction. GPT-4 was provided with similar information and asked to answer the same questions, enabling us to directly compare the performance of artificial intelligence and clinicians. Results: Overall, the clinicians’ average precision (0.7) was higher than that of GPT-4 (0.6) in identifying the SI with plan at intake (n=140) versus no SI (n=200) when using the chief complaint alone, while sensitivity was higher for the GPT-4 (0.62) than the clinicians’ average (0.53). The addition of suicide attempt history increased the clinicians’ average sensitivity (0.59) and precision (0.77) while increasing the GPT-4 sensitivity (0.59) but decreasing the GPT-4 precision (0.54). Performance decreased comparatively when predicting future SI with plan (n=120) versus no SI (n=200) with a chief complaint only for the clinicians (average sensitivity=0.4; average precision=0.59) and the GPT-4 (sensitivity=0.46; precision=0.48). The addition of suicide attempt history increased performance comparatively for the clinicians (average sensitivity=0.46; average precision=0.69) and the GPT-4 (sensitivity=0.74; precision=0.48). Conclusions: GPT-4, with a simple prompt design, produced results on some metrics that approached those of a trained clinician. Additional work must be done before such a model can be piloted in a clinical setting. The model should undergo safety checks for bias, given evidence that LLMs can perpetuate the biases of the underlying data on which they are trained. We believe that LLMs hold promise for augmenting the identification of higher-risk patients at intake and potentially delivering more timely care to patients. Trial Registration:


  • Open Access

    Early Response to Antidepressant Medications in Adults With Major Depressive Disorder

    by Heather G. Belanger, Christine Lee, Zachary Poliacoff, Carina T. Ma Gupta, Mirène Winsberg

    Purpose/Background: Early response after 2 to 4 weeks of antidepressant therapy has been shown to predict remission by 8 to 12 weeks. Most of the work to date on early response has been done using data from randomized controlled trials. Methods/Procedures: This naturalistic study uses archival data from a national tele-mental health company. The positive and negative predictive values as well as sensitivity and specificity were calculated using different drops in baseline Patient Health Questionnaire 9 scores at various periods. Demographic and clinical characteristics were compared between early responders versus those lacking early response. Binary logistic regression analyses determined if early response was predictive of remission, response, and greater than minimal improvement at 14 weeks. For those who do not show early improvement, treatments were investigated using binary logistic regression to see if changes predicted later outcomes. Findings/Results: Positive predictive values for all endpoints improved with the strength of early response but did not improve much with the time allowed for that response to occur. In contrast, negative predictive values increased substantially with time. Using a definition of 30% drop in Patient Health Questionnaire 9 score at week 4, 56.5% of patients were early responders. Early responders were ~3.2 times more likely to achieve remission than those lacking early response. Of nonresponders by week 4, those prescribed atypical antipsychotics (+SSRI) had significantly reduced odds of response at week 14, whereas those prescribed a norepinephrine and dopamine reuptake inhibitor had increased odds. Implications/Conclusions: Early response may be associated with better outcomes at 14 weeks. In those with lack of response by week 4, patients prescribed a norepinephrine and dopamine reuptake inhibitor may achieve superior outcomes.


  • Open Access

    Telehealth-Supported Decision-making Psychiatric Care for Suicidal Ideation: Longitudinal Observational Study

    by Erin O'CAllaghan, Nicole Mahrer, Heather G Belanger, Scott Sullivan, Christine Lee, Carina T Gupta, Mirène Winsberg

    Background: Suicide is a leading cause of death in the United States, and suicidal ideation (SI) is a significant precursor and risk factor for suicide.Objective: This study aimed to examine the impact of a telepsychiatric care platform on changes in SI over time and remission, as well as to investigate the relationship between various demographic and medical factors on SI and SI remission.Methods: Participants included 8581 US-based adults (8366 in the treatment group and 215 in the control group) seeking treatment for depression, anxiety, or both. The treatment group included patients who had completed at least 12 weeks of treatment and had received a prescription for at least one psychiatric medication during the study period. Providers prescribed psychiatric medications for each patient during their first session and received regular data on participants. They also received decision support at treatment onset via the digital platform, which leveraged an empirically derived proprietary precision-prescribing algorithm to give providers real-time care guidelines. Participants in the control group consisted of individuals who completed the initial enrollment data and completed surveys at baseline and 12 weeks but did not receive care.Results: Greater feelings of hopelessness, anhedonia, and feeling bad about oneself were most significantly correlated (r=0.24-0.37) with SI at baseline. Sleep issues and feeling tired or having low energy, although significant, had lower correlations with SI (r=0.13-0.14). In terms of demographic variables, advancing age and education were associated with less SI at baseline (r=−0.16) and 12 weeks (r=−0.10) but less improvement over time (r=−0.12 and −0.11, respectively). Although not different at baseline, the SI expression was evident in 34.4% (74/215) of the participants in the control group and 12.32% (1031/8366) of the participants in the treatment group at 12 weeks. Although the participants in the treatment group improved over time regardless of various demographic variables, participants in the control group with less education worsened over time, after controlling for age and depression severity. A model incorporating the treatment group, age, sex, and 8-item Patient Health Questionnaire scores was 77% accurate in its classification of complete remission. Those in the treatment group were 4.3 times more likely (odds ratio 4.31, 95% CI 2.88-6.44) to have complete SI remission than those in the control group. Female participants and those with advanced education beyond high school were approximately 1.4 times more likely (odds ratio 1.38, 95% CI 1.18-1.62) to remit than their counterparts.Conclusions: The results highlight the efficacy of an antidepressant intervention in reducing SI, in this case administered via a telehealth platform and with decision support, as well as the importance of considering covariates, or subpopulations, when considering SI. Further research and refinement, ideally via randomized controlled trials, are needed.


  • Big data and targeted machine learning in action to assist medical decision in the ICU

    by Romain Pirracchio, Mitchell J Cohen, Ivana Malenica, Jonathan Cohen, Antoine Chambaz, Maxime Cannesson, Christine Lee, Matthieu Resche-Rigon, Alan Hubbard