Jonathan Nebeker
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Jonathan Nebeker has a total of 71 co-authors. Here are the top co-authors by number of shared publications:
Charles L. Bennett
4 shared publications- Peter L. Elkin
Department of Biomedical Informatics, University at Buffalo
4 shared publications - Steven H. Brown
Office of Health Informatics, Department of Veterans Affairs
4 shared publications - Angela Laurio
Department of Veterans Affairs, Washington, DC, USA
3 shared publications Jennifer M. Hoffman
Salt Lake Informatics, Decision Enhancement, and Surveillance (IDEAS) Center, Salt Lake City
3 shared publications- Merry Ward
Veterans Health Administration Office of Health Informatics Department of Veterans Affairs Washington DC, DC United States
3 shared publications
Latest Publications
- Open Access
- 12 May 2025
- Published by IOS Press
- Vol. 326, 2-6
Retrieval Augmented Generation: What Works and Lessons Learned
Retrieval Augmented Generation has been shown to improve the output of large language models (LLMs) by providing context to the question or scenario posed to the model. We have tried a series of experiments to understand how best to improve the performance of the native models. We present the results of each of several experiments. These can serve as lessons learned for scientists looking to improve the performance of large language models for medical question answering tasks.
- 12 May 2025
- Published by IOS Press
- Vol. 326, 2-6
Open AccessRetrieval Augmented Generation: What Works and Lessons Learned
Retrieval Augmented Generation has been shown to improve the output of large language models (LLMs) by providing context to the question or scenario posed to the model. We have tried a series of experiments to understand how best to improve the performance of the native models. We present the results of each of several experiments. These can serve as lessons learned for scientists looking to improve the performance of large language models for medical question answering tasks.
- Open Access
- 12 May 2025
- Published by IOS Press
- Vol. 326, 22-26
The Semantic Clinical Artificial Intelligence (SCAI) Chatbot: Preliminary Usability Testing
We developed a doctor-facing chatbot named SCAI. We aimed to evaluate response speed and user satisfaction. Ten questions were used to test the response speed of the SCAI chatbot. The response time was long, but overall the participants were satisfied with SCAI.
- 12 May 2025
- Published by IOS Press
- Vol. 326, 22-26
Open AccessThe Semantic Clinical Artificial Intelligence (SCAI) Chatbot: Preliminary Usability Testing
We developed a doctor-facing chatbot named SCAI. We aimed to evaluate response speed and user satisfaction. Ten questions were used to test the response speed of the SCAI chatbot. The response time was long, but overall the participants were satisfied with SCAI.
- Open Access
- 30 April 2024
- Published by JMIR Publications Inc. in JMIR Public Health and Surveillance
- Vol. 10, e49841
Defining the Subtypes of Long COVID and Risk Factors for Prolonged Disease: Population-Based Case-Crossover Study
Background: There have been over 772 million confirmed cases of COVID-19 worldwide. A significant portion of these infections will lead to long COVID (post–COVID-19 condition) and its attendant morbidities and costs. Numerous life-altering complications have already been associated with the development of long COVID, including chronic fatigue, brain fog, and dangerous heart rhythms. Objective: We aim to derive an actionable long COVID case definition consisting of significantly increased signs, symptoms, and diagnoses to support pandemic-related clinical, public health, research, and policy initiatives. Methods: This research employs a case-crossover population-based study using International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) data generated at Veterans Affairs medical centers nationwide between January 1, 2020, and August 18, 2022. In total, 367,148 individuals with ICD-10-CM data both before and after a positive COVID-19 test were selected for analysis. We compared ICD-10-CM codes assigned 1 to 7 months following each patient’s positive test with those assigned up to 6 months prior. Further, 350,315 patients had novel codes assigned during this window of time. We defined signs, symptoms, and diagnoses as being associated with long COVID if they had a novel case frequency of ≥1:1000, and they significantly increased in our entire cohort after a positive test. We present odds ratios with CIs for long COVID signs, symptoms, and diagnoses, organized by ICD-10-CM functional groups and medical specialty. We used our definition to assess long COVID risk based on a patient’s demographics, Elixhauser score, vaccination status, and COVID-19 disease severity. Results: We developed a long COVID definition consisting of 323 ICD-10-CM diagnosis codes grouped into 143 ICD-10-CM functional groups that were significantly increased in our 367,148 patient post–COVID-19 population. We defined 17 medical-specialty long COVID subtypes such as cardiology long COVID. Patients who were COVID-19–positive developed signs, symptoms, or diagnoses included in our long COVID definition at a proportion of at least 59.7% (268,320/449,450, based on a denominator of all patients who were COVID-19–positive). The long COVID cohort was 8 years older with more comorbidities (2-year Elixhauser score 7.97 in the patients with long COVID vs 4.21 in the patients with non–long COVID). Patients who had a more severe bout of COVID-19, as judged by their minimum oxygen saturation level, were also more likely to develop long COVID. Conclusions: An actionable, data-driven definition of long COVID can help clinicians screen for and diagnose long COVID, allowing identified patients to be admitted into appropriate monitoring and treatment programs. This long COVID definition can also support public health, research, and policy initiatives. Patients with COVID-19 who are older or have low oxygen saturation levels during their bout of COVID-19, or those who have multiple comorbidities should be preferentially watched for the development of long COVID.
- 30 April 2024
- Published by JMIR Publications Inc. in JMIR Public Health and Surveillance
- Vol. 10, e49841
Open AccessDefining the Subtypes of Long COVID and Risk Factors for Prolonged Disease: Population-Based Case-Crossover Study
Background: There have been over 772 million confirmed cases of COVID-19 worldwide. A significant portion of these infections will lead to long COVID (post–COVID-19 condition) and its attendant morbidities and costs. Numerous life-altering complications have already been associated with the development of long COVID, including chronic fatigue, brain fog, and dangerous heart rhythms. Objective: We aim to derive an actionable long COVID case definition consisting of significantly increased signs, symptoms, and diagnoses to support pandemic-related clinical, public health, research, and policy initiatives. Methods: This research employs a case-crossover population-based study using International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) data generated at Veterans Affairs medical centers nationwide between January 1, 2020, and August 18, 2022. In total, 367,148 individuals with ICD-10-CM data both before and after a positive COVID-19 test were selected for analysis. We compared ICD-10-CM codes assigned 1 to 7 months following each patient’s positive test with those assigned up to 6 months prior. Further, 350,315 patients had novel codes assigned during this window of time. We defined signs, symptoms, and diagnoses as being associated with long COVID if they had a novel case frequency of ≥1:1000, and they significantly increased in our entire cohort after a positive test. We present odds ratios with CIs for long COVID signs, symptoms, and diagnoses, organized by ICD-10-CM functional groups and medical specialty. We used our definition to assess long COVID risk based on a patient’s demographics, Elixhauser score, vaccination status, and COVID-19 disease severity. Results: We developed a long COVID definition consisting of 323 ICD-10-CM diagnosis codes grouped into 143 ICD-10-CM functional groups that were significantly increased in our 367,148 patient post–COVID-19 population. We defined 17 medical-specialty long COVID subtypes such as cardiology long COVID. Patients who were COVID-19–positive developed signs, symptoms, or diagnoses included in our long COVID definition at a proportion of at least 59.7% (268,320/449,450, based on a denominator of all patients who were COVID-19–positive). The long COVID cohort was 8 years older with more comorbidities (2-year Elixhauser score 7.97 in the patients with long COVID vs 4.21 in the patients with non–long COVID). Patients who had a more severe bout of COVID-19, as judged by their minimum oxygen saturation level, were also more likely to develop long COVID. Conclusions: An actionable, data-driven definition of long COVID can help clinicians screen for and diagnose long COVID, allowing identified patients to be admitted into appropriate monitoring and treatment programs. This long COVID definition can also support public health, research, and policy initiatives. Patients with COVID-19 who are older or have low oxygen saturation levels during their bout of COVID-19, or those who have multiple comorbidities should be preferentially watched for the development of long COVID.
- Open Access
- 6 February 2024
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 150, 104605
EHR-BERT: A BERT-based model for effective anomaly detection in electronic health records
- 6 February 2024
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 150, 104605
Open AccessEHR-BERT: A BERT-based model for effective anomaly detection in electronic health records
- Open Access
- 11 June 2023
- Published by JMIR Publications Inc. in JMIR Preprints
Defining the Subtypes of Long COVID and Risk Factors for Prolonged Disease: Population-Based Case-Crossover Study (Preprint)
BACKGROUND There have been over 772 million confirmed cases of COVID-19 worldwide. A significant portion of these infections will lead to long COVID (post–COVID-19 condition) and its attendant morbidities and costs. Numerous life-altering complications have already been associated with the development of long COVID, including chronic fatigue, brain fog, and dangerous heart rhythms. OBJECTIVE We aim to derive an actionable long COVID case definition consisting of significantly increased signs, symptoms, and diagnoses to support pandemic-related clinical, public health, research, and policy initiatives. METHODS This research employs a case-crossover population-based study using International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) data generated at Veterans Affairs medical centers nationwide between January 1, 2020, and August 18, 2022. In total, 367,148 individuals with ICD-10-CM data both before and after a positive COVID-19 test were selected for analysis. We compared ICD-10-CM codes assigned 1 to 7 months following each patient’s positive test with those assigned up to 6 months prior. Further, 350,315 patients had novel codes assigned during this window of time. We defined signs, symptoms, and diagnoses as being associated with long COVID if they had a novel case frequency of ≥1:1000, and they significantly increased in our entire cohort after a positive test. We present odds ratios with CIs for long COVID signs, symptoms, and diagnoses, organized by ICD-10-CM functional groups and medical specialty. We used our definition to assess long COVID risk based on a patient’s demographics, Elixhauser score, vaccination status, and COVID-19 disease severity. RESULTS We developed a long COVID definition consisting of 323 ICD-10-CM diagnosis codes grouped into 143 ICD-10-CM functional groups that were significantly increased in our 367,148 patient post–COVID-19 population. We defined 17 medical-specialty long COVID subtypes such as cardiology long COVID. Patients who were COVID-19–positive developed signs, symptoms, or diagnoses included in our long COVID definition at a proportion of at least 59.7% (268,320/449,450, based on a denominator of all patients who were COVID-19–positive). The long COVID cohort was 8 years older with more comorbidities (2-year Elixhauser score 7.97 in the patients with long COVID vs 4.21 in the patients with non–long COVID). Patients who had a more severe bout of COVID-19, as judged by their minimum oxygen saturation level, were also more likely to develop long COVID. CONCLUSIONS An actionable, data-driven definition of long COVID can help clinicians screen for and diagnose long COVID, allowing identified patients to be admitted into appropriate monitoring and treatment programs. This long COVID definition can also support public health, research, and policy initiatives. Patients with COVID-19 who are older or have low oxygen saturation levels during their bout of COVID-19, or those who have multiple comorbidities should be preferentially watched for the development of long COVID.
- 11 June 2023
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessDefining the Subtypes of Long COVID and Risk Factors for Prolonged Disease: Population-Based Case-Crossover Study (Preprint)
BACKGROUND There have been over 772 million confirmed cases of COVID-19 worldwide. A significant portion of these infections will lead to long COVID (post–COVID-19 condition) and its attendant morbidities and costs. Numerous life-altering complications have already been associated with the development of long COVID, including chronic fatigue, brain fog, and dangerous heart rhythms. OBJECTIVE We aim to derive an actionable long COVID case definition consisting of significantly increased signs, symptoms, and diagnoses to support pandemic-related clinical, public health, research, and policy initiatives. METHODS This research employs a case-crossover population-based study using International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) data generated at Veterans Affairs medical centers nationwide between January 1, 2020, and August 18, 2022. In total, 367,148 individuals with ICD-10-CM data both before and after a positive COVID-19 test were selected for analysis. We compared ICD-10-CM codes assigned 1 to 7 months following each patient’s positive test with those assigned up to 6 months prior. Further, 350,315 patients had novel codes assigned during this window of time. We defined signs, symptoms, and diagnoses as being associated with long COVID if they had a novel case frequency of ≥1:1000, and they significantly increased in our entire cohort after a positive test. We present odds ratios with CIs for long COVID signs, symptoms, and diagnoses, organized by ICD-10-CM functional groups and medical specialty. We used our definition to assess long COVID risk based on a patient’s demographics, Elixhauser score, vaccination status, and COVID-19 disease severity. RESULTS We developed a long COVID definition consisting of 323 ICD-10-CM diagnosis codes grouped into 143 ICD-10-CM functional groups that were significantly increased in our 367,148 patient post–COVID-19 population. We defined 17 medical-specialty long COVID subtypes such as cardiology long COVID. Patients who were COVID-19–positive developed signs, symptoms, or diagnoses included in our long COVID definition at a proportion of at least 59.7% (268,320/449,450, based on a denominator of all patients who were COVID-19–positive). The long COVID cohort was 8 years older with more comorbidities (2-year Elixhauser score 7.97 in the patients with long COVID vs 4.21 in the patients with non–long COVID). Patients who had a more severe bout of COVID-19, as judged by their minimum oxygen saturation level, were also more likely to develop long COVID. CONCLUSIONS An actionable, data-driven definition of long COVID can help clinicians screen for and diagnose long COVID, allowing identified patients to be admitted into appropriate monitoring and treatment programs. This long COVID definition can also support public health, research, and policy initiatives. Patients with COVID-19 who are older or have low oxygen saturation levels during their bout of COVID-19, or those who have multiple comorbidities should be preferentially watched for the development of long COVID.
- Open Access
- 13 October 2022
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 135, 104219
Detecting anomalous sequences in electronic health records using higher-order tensor networks
- 13 October 2022
- Published by Elsevier in Journal of Biomedical Informatics
- Vol. 135, 104219
Open AccessDetecting anomalous sequences in electronic health records using higher-order tensor networks
- Open Access
- 20 May 2022
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 10 (5) , e32168
Integrated Health Record Viewers and Reduction in Duplicate Medical Imaging: Retrospective Observational Analysis
Background: Health information exchange and multiplatform health record viewers support more informed medical decisions, improve quality of care, and reduce the risk of adverse outcomes due to fragmentation and discontinuity in care during transition of care. An example of a multiplatform health record viewer is the VA/DoD Joint Longitudinal Viewer (JLV), which supports the Department of Veterans Affairs (VA) and Department of Defense (DoD) health care providers with read-only access to patient medical records integrated from multiple sources. JLV is intended to support more informed medical decisions such as reducing duplicate medical imaging when previous image study results may meet current clinical needs. Objective: We estimated the impact of provider usage of JLV on duplicate imaging for service members transitioning from the DoD to the VA health care system. Methods: We conducted a retrospective cross-sectional study in fiscal year 2018 to examine the relationship between providers’ use of JLV and the likelihood of ordering duplicate images. Our sample included recently separated service members who had a VA primary care visit in fiscal year 2018 within 90 days of a DoD imaging study. Patients who received at least one imaging study at VA within 90 days of a DoD imaging study of the same imaging mode and on the same body part are considered to have received potentially duplicate imaging studies. We use a logistic regression model with “JLV provider” (providers with 1 or more JLV audits in the prior 6 months) as the independent variable to estimate the relationship between JLV use and ordering of duplicate images. Control variables included provider image ordering rates in the prior 6 months, provider type, patient demographics (age, race, gender), and clinical characteristics (Elixhauser comorbidity score). Results: Providers known to utilize JLV in the prior 6 months order fewer duplicate images relative to providers not utilizing JLV for similar visits over time (odds ratio 0.44, 95% CI 0.24-0.78; P=.005). This effect is robust across multiple specifications of linear and logistic regression models. The provider’s practice pattern of ordering image studies and the patient’s health status are powerful confounders. Conclusions: This study provides evidence that adoption of a longitudinal viewer of health records from multiple electronic health record systems is associated with a reduced likelihood of ordering duplicate images. Investments in health information exchange systems may be effective ways to improve the quality of care and reduce adverse outcomes for patients experiencing fragmentation and discontinuity of care.
- 20 May 2022
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 10 (5) , e32168
Open AccessIntegrated Health Record Viewers and Reduction in Duplicate Medical Imaging: Retrospective Observational Analysis
Background: Health information exchange and multiplatform health record viewers support more informed medical decisions, improve quality of care, and reduce the risk of adverse outcomes due to fragmentation and discontinuity in care during transition of care. An example of a multiplatform health record viewer is the VA/DoD Joint Longitudinal Viewer (JLV), which supports the Department of Veterans Affairs (VA) and Department of Defense (DoD) health care providers with read-only access to patient medical records integrated from multiple sources. JLV is intended to support more informed medical decisions such as reducing duplicate medical imaging when previous image study results may meet current clinical needs. Objective: We estimated the impact of provider usage of JLV on duplicate imaging for service members transitioning from the DoD to the VA health care system. Methods: We conducted a retrospective cross-sectional study in fiscal year 2018 to examine the relationship between providers’ use of JLV and the likelihood of ordering duplicate images. Our sample included recently separated service members who had a VA primary care visit in fiscal year 2018 within 90 days of a DoD imaging study. Patients who received at least one imaging study at VA within 90 days of a DoD imaging study of the same imaging mode and on the same body part are considered to have received potentially duplicate imaging studies. We use a logistic regression model with “JLV provider” (providers with 1 or more JLV audits in the prior 6 months) as the independent variable to estimate the relationship between JLV use and ordering of duplicate images. Control variables included provider image ordering rates in the prior 6 months, provider type, patient demographics (age, race, gender), and clinical characteristics (Elixhauser comorbidity score). Results: Providers known to utilize JLV in the prior 6 months order fewer duplicate images relative to providers not utilizing JLV for similar visits over time (odds ratio 0.44, 95% CI 0.24-0.78; P=.005). This effect is robust across multiple specifications of linear and logistic regression models. The provider’s practice pattern of ordering image studies and the patient’s health status are powerful confounders. Conclusions: This study provides evidence that adoption of a longitudinal viewer of health records from multiple electronic health record systems is associated with a reduced likelihood of ordering duplicate images. Investments in health information exchange systems may be effective ways to improve the quality of care and reduce adverse outcomes for patients experiencing fragmentation and discontinuity of care.
- Open Access
- 16 July 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Integrated Health Record Viewers and Reduction in Duplicate Medical Imaging: Retrospective Observational Analysis (Preprint)
BACKGROUND Health information exchange and multiplatform health record viewers support more informed medical decisions, improve quality of care, and reduce the risk of adverse outcomes due to fragmentation and discontinuity in care during transition of care. An example of a multiplatform health record viewer is the VA/DoD Joint Longitudinal Viewer (JLV), which supports the Department of Veterans Affairs (VA) and Department of Defense (DoD) health care providers with read-only access to patient medical records integrated from multiple sources. JLV is intended to support more informed medical decisions such as reducing duplicate medical imaging when previous image study results may meet current clinical needs. OBJECTIVE We estimated the impact of provider usage of JLV on duplicate imaging for service members transitioning from the DoD to the VA health care system. METHODS We conducted a retrospective cross-sectional study in fiscal year 2018 to examine the relationship between providers’ use of JLV and the likelihood of ordering duplicate images. Our sample included recently separated service members who had a VA primary care visit in fiscal year 2018 within 90 days of a DoD imaging study. Patients who received at least one imaging study at VA within 90 days of a DoD imaging study of the same imaging mode and on the same body part are considered to have received potentially duplicate imaging studies. We use a logistic regression model with “JLV provider” (providers with 1 or more JLV audits in the prior 6 months) as the independent variable to estimate the relationship between JLV use and ordering of duplicate images. Control variables included provider image ordering rates in the prior 6 months, provider type, patient demographics (age, race, gender), and clinical characteristics (Elixhauser comorbidity score). RESULTS Providers known to utilize JLV in the prior 6 months order fewer duplicate images relative to providers not utilizing JLV for similar visits over time (odds ratio 0.44, 95% CI 0.24-0.78; P=.005). This effect is robust across multiple specifications of linear and logistic regression models. The provider’s practice pattern of ordering image studies and the patient’s health status are powerful confounders. CONCLUSIONS This study provides evidence that adoption of a longitudinal viewer of health records from multiple electronic health record systems is associated with a reduced likelihood of ordering duplicate images. Investments in health information exchange systems may be effective ways to improve the quality of care and reduce adverse outcomes for patients experiencing fragmentation and discontinuity of care.
- 16 July 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessIntegrated Health Record Viewers and Reduction in Duplicate Medical Imaging: Retrospective Observational Analysis (Preprint)
BACKGROUND Health information exchange and multiplatform health record viewers support more informed medical decisions, improve quality of care, and reduce the risk of adverse outcomes due to fragmentation and discontinuity in care during transition of care. An example of a multiplatform health record viewer is the VA/DoD Joint Longitudinal Viewer (JLV), which supports the Department of Veterans Affairs (VA) and Department of Defense (DoD) health care providers with read-only access to patient medical records integrated from multiple sources. JLV is intended to support more informed medical decisions such as reducing duplicate medical imaging when previous image study results may meet current clinical needs. OBJECTIVE We estimated the impact of provider usage of JLV on duplicate imaging for service members transitioning from the DoD to the VA health care system. METHODS We conducted a retrospective cross-sectional study in fiscal year 2018 to examine the relationship between providers’ use of JLV and the likelihood of ordering duplicate images. Our sample included recently separated service members who had a VA primary care visit in fiscal year 2018 within 90 days of a DoD imaging study. Patients who received at least one imaging study at VA within 90 days of a DoD imaging study of the same imaging mode and on the same body part are considered to have received potentially duplicate imaging studies. We use a logistic regression model with “JLV provider” (providers with 1 or more JLV audits in the prior 6 months) as the independent variable to estimate the relationship between JLV use and ordering of duplicate images. Control variables included provider image ordering rates in the prior 6 months, provider type, patient demographics (age, race, gender), and clinical characteristics (Elixhauser comorbidity score). RESULTS Providers known to utilize JLV in the prior 6 months order fewer duplicate images relative to providers not utilizing JLV for similar visits over time (odds ratio 0.44, 95% CI 0.24-0.78; P=.005). This effect is robust across multiple specifications of linear and logistic regression models. The provider’s practice pattern of ordering image studies and the patient’s health status are powerful confounders. CONCLUSIONS This study provides evidence that adoption of a longitudinal viewer of health records from multiple electronic health record systems is associated with a reduced likelihood of ordering duplicate images. Investments in health information exchange systems may be effective ways to improve the quality of care and reduce adverse outcomes for patients experiencing fragmentation and discontinuity of care.
- Open Access
- 1 May 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-4
Topic Modeling to Discern Irregular Order Patterns in Unlabeled Electronic Health Records
This paper presents an application of topic modeling on event sequences of Electronic Health Record (EHR) orders. Metaphorically, we approach clinical order event sequences of unlabeled data as if they are documents where words are the events that occurred in the history of an order. We demonstrate the approach leveraging Consult orders data. The details of the data preprocessing and the data structures are described along with the data sources. Latent Dirichlet Allocation (LDA) is leveraged to fit against the limited datasets prepared. Another open source tool - LDAvis is used for exploratory analysis of the LDA results. The preliminary results revealed some order patterns that are qualitatively evaluated as potential irregular transitions. The goal of this analysis is to provide unsupervised learning application to domain experts in the absence of labeled data where they can investigate captured patterns and identify irregular transitions of orders. Ultimately, such efforts will guide formalization of hazard detection algorithms that monitor EHR data to identify health information technology related hazards.
- 1 May 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-4
Open AccessTopic Modeling to Discern Irregular Order Patterns in Unlabeled Electronic Health Records
This paper presents an application of topic modeling on event sequences of Electronic Health Record (EHR) orders. Metaphorically, we approach clinical order event sequences of unlabeled data as if they are documents where words are the events that occurred in the history of an order. We demonstrate the approach leveraging Consult orders data. The details of the data preprocessing and the data structures are described along with the data sources. Latent Dirichlet Allocation (LDA) is leveraged to fit against the limited datasets prepared. Another open source tool - LDAvis is used for exploratory analysis of the LDA results. The preliminary results revealed some order patterns that are qualitatively evaluated as potential irregular transitions. The goal of this analysis is to provide unsupervised learning application to domain experts in the absence of labeled data where they can investigate captured patterns and identify irregular transitions of orders. Ultimately, such efforts will guide formalization of hazard detection algorithms that monitor EHR data to identify health information technology related hazards.
- 1 June 2011
- Published by Wolters Kluwer Health in Journal of Patient Safety
- Vol. 7 (2) , 66-71
Consensus Building for Development of Outpatient Adverse Drug Event Triggers
Objectives: Adverse drug event (ADE) detection is an important priority of patient safety research. Trigger tools have been developed to help identify ADEs. As part of a larger study, we developed complex and specific trigger algorithms intended for concurrent use with clinical care to detect outpatient ADEs. This article assesses the use of a modified Delphi process to obtain expert consensus on the value of these triggers. Methods: We selected a panel of distinguished clinical and research experts to participate in the modified Delphi process. We created a set of outpatient ADE triggers based on literature review, clinical input, and methodological expertise. The importance of the targeted ADEs, associated drug classes, and trigger logic was used to rate each trigger. Specific criteria were developed to establish consensus. Results: The modified Delphi process established consensus on 6 outpatient ADE triggers to test with patient-level data based on high ratings of utility for patient-level interventions. These triggers focused on detecting ADEs caused by the following drugs or drug classes: bone marrow toxins, potassium raisers, potassium reducers, creatinine, warfarin, and sedative hypnotics. Participants reported including all aspects of the trigger in their ratings, despite our efforts to separate evaluation of clinical need and trigger logic. Participants' expertise affected the evaluation of trigger rules, leading to contradictory feedback on how to improve trigger design. Conclusions: The efficiency of the modified Delphi method could be improved by allowing participants to produce an overall summary score that incorporates both the clinical value and the general logic of the trigger. Revising and improving trigger design should be conducted in a separate process limited only to trigger experts.
- 1 June 2011
- Published by Wolters Kluwer Health in Journal of Patient Safety
- Vol. 7 (2) , 66-71
Consensus Building for Development of Outpatient Adverse Drug Event Triggers
Objectives: Adverse drug event (ADE) detection is an important priority of patient safety research. Trigger tools have been developed to help identify ADEs. As part of a larger study, we developed complex and specific trigger algorithms intended for concurrent use with clinical care to detect outpatient ADEs. This article assesses the use of a modified Delphi process to obtain expert consensus on the value of these triggers. Methods: We selected a panel of distinguished clinical and research experts to participate in the modified Delphi process. We created a set of outpatient ADE triggers based on literature review, clinical input, and methodological expertise. The importance of the targeted ADEs, associated drug classes, and trigger logic was used to rate each trigger. Specific criteria were developed to establish consensus. Results: The modified Delphi process established consensus on 6 outpatient ADE triggers to test with patient-level data based on high ratings of utility for patient-level interventions. These triggers focused on detecting ADEs caused by the following drugs or drug classes: bone marrow toxins, potassium raisers, potassium reducers, creatinine, warfarin, and sedative hypnotics. Participants reported including all aspects of the trigger in their ratings, despite our efforts to separate evaluation of clinical need and trigger logic. Participants' expertise affected the evaluation of trigger rules, leading to contradictory feedback on how to improve trigger design. Conclusions: The efficiency of the modified Delphi method could be improved by allowing participants to produce an overall summary score that incorporates both the clinical value and the general logic of the trigger. Revising and improving trigger design should be conducted in a separate process limited only to trigger experts.