Anthony B. Cardillo
Anthony B. Cardillo has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
- Abraham Aizer Brody
Rory Meyers College of Nursing , New York University, New York, NY 10010
2 shared publications - Alba Avoricani
Affiliation NYU Grossman School of Medicine, New York, New York, United States of America
2 shared publications - Andrew J. Clayburn
Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, Minnesota, United States of America
2 shared publications - Arjun Vijay Masurkar
Department of Neurology, NYU Grossman School of Medicine, 145 East 32nd Street, New York, NY 10016, USA
2 shared publications - Benjamin A. Silva
Affiliation NYU Grossman School of Medicine, New York, New York, United States of America
2 shared publications - Eesha Chakravartty
Affiliation NYU Grossman School of Medicine, New York, New York, United States of America
2 shared publications
Latest Publications
- Open Access
- 11 December 2024
- Published by Public Library of Science (PLoS) in PLOS Digital Health
- Vol. 3 (12) , e0000685
Evaluating Large Language Models in extracting cognitive exam dates and scores
Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
- 11 December 2024
- Published by Public Library of Science (PLoS) in PLOS Digital Health
- Vol. 3 (12) , e0000685
Open AccessEvaluating Large Language Models in extracting cognitive exam dates and scores
Ensuring reliability of Large Language Models (LLMs) in clinical tasks is crucial. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
- 16 October 2024
- Published by Georg Thieme Verlag KG in Applied Clinical Informatics
- Vol. 15 (05) , 1093-1096
Reference Ranges for All: Implementing Reference Ranges for Transgender and Nonbinary Patients
Objectives This study aimed to highlight the necessity of developing and implementing appropriate reference ranges for transgender and nonbinary (TGNB) patient populations to minimize misinterpretation of laboratory results and ensure equitable health care. Case Report We describe a situation where a TGNB patient's abnormal laboratory values were not flagged due to undefined reference ranges for gender “X” in the Laboratory Information System (LIS). Implementation of additional reference ranges mapped to sex label “X” showed significant improvement in flagging abnormal lab results, utilizing sex-invariant reporting as an interim solution while monitoring developments on TGNB-specific reference ranges. Conclusion Informatics professionals should assess their institution's policies for registration and lab reporting on TGNB patients as nonimplementation poses significant patient safety risks. Best practices include using TGNB-specific reference ranges emerging in the literature, reporting both male and female reference ranges for clinical interpretation and sex-invariant reporting.
- 16 October 2024
- Published by Georg Thieme Verlag KG in Applied Clinical Informatics
- Vol. 15 (05) , 1093-1096
Reference Ranges for All: Implementing Reference Ranges for Transgender and Nonbinary Patients
Objectives This study aimed to highlight the necessity of developing and implementing appropriate reference ranges for transgender and nonbinary (TGNB) patient populations to minimize misinterpretation of laboratory results and ensure equitable health care. Case Report We describe a situation where a TGNB patient's abnormal laboratory values were not flagged due to undefined reference ranges for gender “X” in the Laboratory Information System (LIS). Implementation of additional reference ranges mapped to sex label “X” showed significant improvement in flagging abnormal lab results, utilizing sex-invariant reporting as an interim solution while monitoring developments on TGNB-specific reference ranges. Conclusion Informatics professionals should assess their institution's policies for registration and lab reporting on TGNB patients as nonimplementation poses significant patient safety risks. Best practices include using TGNB-specific reference ranges emerging in the literature, reporting both male and female reference ranges for clinical interpretation and sex-invariant reporting.
- Open Access
Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores
Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
Open AccessEvaluating Large Language Models in Extracting Cognitive Exam Dates and Scores
Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss’ Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
- Open Access
- 3 May 2023
- Published by Wolters Kluwer Health in Annals of Medicine & Surgery
- Vol. 85 (6) , 2640-2646
Exploration of HER2 (ERBB2) immunohistochemistry in non-small cell lung cancer: correlation with ERBB2 mutational status: experimental research
ERBB2 (HER2) is a gene in humans that encodes the ERBB2 protein, a member of the epidermal growth factor receptor family. Non-small cell lung carcinomas do not commonly harbour ERBB2 mutations, with clinical trials conducted to assess for targeted response and progression-free survival. We retrieved cases of lung adenocarcinoma with next-generation sequencing proven ERBB2 point mutations (n=8) or amplifications (n=11) and assessed the concordance of commercially available ERBB2 (HER2) immunohistochemical antibodies with the next-generation sequencing result. At present, no commercially available ERBB2 clone can accurately detect ERBB2 mutations consistently in non-small cell lung carcinoma specimens, but amplifications can be detected with reasonable diagnostic accuracy.
- 3 May 2023
- Published by Wolters Kluwer Health in Annals of Medicine & Surgery
- Vol. 85 (6) , 2640-2646
Open AccessExploration of HER2 (ERBB2) immunohistochemistry in non-small cell lung cancer: correlation with ERBB2 mutational status: experimental research
ERBB2 (HER2) is a gene in humans that encodes the ERBB2 protein, a member of the epidermal growth factor receptor family. Non-small cell lung carcinomas do not commonly harbour ERBB2 mutations, with clinical trials conducted to assess for targeted response and progression-free survival. We retrieved cases of lung adenocarcinoma with next-generation sequencing proven ERBB2 point mutations (n=8) or amplifications (n=11) and assessed the concordance of commercially available ERBB2 (HER2) immunohistochemical antibodies with the next-generation sequencing result. At present, no commercially available ERBB2 clone can accurately detect ERBB2 mutations consistently in non-small cell lung carcinoma specimens, but amplifications can be detected with reasonable diagnostic accuracy.
- Open Access
- 15 June 2022
- Published by American Association for Cancer Research (AACR) in Cancer Research
- Vol. 82 (12_Supplem) , 4088
Abstract 4088: A novel real world clinico-genomics data platform identifies breast cancer histologic and molecular characteristics as candidate predictors for time to distant metastasis
Introduction: Currently, the NCCN guidelines for invasive breast cancer (BC) surveillance are the same regardless of tumor pathology. The length of time between initial diagnosis and documented distant metastasis based on tumor subtypes has not been well-studied in real-world settings, which means that for any given patient, generalized screening recommendations may be either too stringent or too relaxed. Here, we report how tumor histologic and molecular characteristics analyzed from a real-world data platform correlate with the average time to distant recurrence. This information can guide more personalized surveillance for BC patients. Methods: Ciitizen (a wholly owned subsidiary of Invitae) leverages HIPAA right of access on behalf of patients to collect their medical records, extracts relevant clinical information into a standardized and validated longitudinal dataset, and - with their consent - enables research use of this data. From our current invasive BC cohort (n=1011), we identified a subcohort who were diagnosed at Stage I-III with associated histology and/or molecular characterization, then ultimately went on to develop distant metastases (n=195 cases involving 321 sites: bone, brain, liver, and/or lung). We calculated time from initial diagnosis to confirmation of distant metastasis at each site, and statistically analyzed differences using pooled hypothesis testing. Results: In addition to finding expected associations between molecular type and site(s) of metastasis, i.e. bone and brain more common in Hormone Receptor Positive (HR+) subtypes vs HR-, we also characterized the average time to distant metastasis (TDM) to identify other statistically significant associations: Molecular Type: HR-/HER2- (n=30) has an average TDM of 1124 days, which is significantly faster than HR+/HER2- (2224 days; n=197) and HR+/HER2+ (2159 days; n=32) (p=0.0043). BRCA: Individuals with BRCA1/2 pathogenic germline variants have an average TDM of 1043 days (n=15), significantly earlier than individuals with normal BRCA1/2 (2238 days; n=179) (p Citation Format: Amanda Nottke, Anthony B. Cardillo, Hana E. Littleford, Lisandra West-Odell, Alexandra Berk. A novel real world clinico-genomics data platform identifies breast cancer histologic and molecular characteristics as candidate predictors for time to distant metastasis [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 4088.
- 15 June 2022
- Published by American Association for Cancer Research (AACR) in Cancer Research
- Vol. 82 (12_Supplem) , 4088
Open AccessAbstract 4088: A novel real world clinico-genomics data platform identifies breast cancer histologic and molecular characteristics as candidate predictors for time to distant metastasis
Introduction: Currently, the NCCN guidelines for invasive breast cancer (BC) surveillance are the same regardless of tumor pathology. The length of time between initial diagnosis and documented distant metastasis based on tumor subtypes has not been well-studied in real-world settings, which means that for any given patient, generalized screening recommendations may be either too stringent or too relaxed. Here, we report how tumor histologic and molecular characteristics analyzed from a real-world data platform correlate with the average time to distant recurrence. This information can guide more personalized surveillance for BC patients. Methods: Ciitizen (a wholly owned subsidiary of Invitae) leverages HIPAA right of access on behalf of patients to collect their medical records, extracts relevant clinical information into a standardized and validated longitudinal dataset, and - with their consent - enables research use of this data. From our current invasive BC cohort (n=1011), we identified a subcohort who were diagnosed at Stage I-III with associated histology and/or molecular characterization, then ultimately went on to develop distant metastases (n=195 cases involving 321 sites: bone, brain, liver, and/or lung). We calculated time from initial diagnosis to confirmation of distant metastasis at each site, and statistically analyzed differences using pooled hypothesis testing. Results: In addition to finding expected associations between molecular type and site(s) of metastasis, i.e. bone and brain more common in Hormone Receptor Positive (HR+) subtypes vs HR-, we also characterized the average time to distant metastasis (TDM) to identify other statistically significant associations: Molecular Type: HR-/HER2- (n=30) has an average TDM of 1124 days, which is significantly faster than HR+/HER2- (2224 days; n=197) and HR+/HER2+ (2159 days; n=32) (p=0.0043). BRCA: Individuals with BRCA1/2 pathogenic germline variants have an average TDM of 1043 days (n=15), significantly earlier than individuals with normal BRCA1/2 (2238 days; n=179) (p Citation Format: Amanda Nottke, Anthony B. Cardillo, Hana E. Littleford, Lisandra West-Odell, Alexandra Berk. A novel real world clinico-genomics data platform identifies breast cancer histologic and molecular characteristics as candidate predictors for time to distant metastasis [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 4088.