Yizhao Ni
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Yizhao Ni has a total of 435 co-authors. Here are the top co-authors by number of shared publications:
- Kristin Melton
Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OH, USA. Division of Neonatology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, USA
10 shared publications - Todd Lingren
Division of Biomedical Informatics Cincinnati Children's Hospital Medical Center Cincinnati, OH United States
9 shared publications - Eric Steven Kirkendall
University of Cincinnati College of Medicine
8 shared publications Imre Solti
Department of Biomedical Informatics
8 shared publicationsTodd Lingren
8 shared publicationsAlycia Bachtel
Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA
4 shared publications
Latest Publications
- Open Access
- 16 December 2022
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 10 (12) , e37833
Implementation of Machine Learning Pipelines for Clinical Practice: Development and Validation Study
Background: Artificial intelligence (AI) technologies, such as machine learning and natural language processing, have the potential to provide new insights into complex health data. Although powerful, these algorithms rarely move from experimental studies to direct clinical care implementation. Objective: We aimed to describe the key components for successful development and integration of two AI technology–based research pipelines for clinical practice. Methods: We summarized the approach, results, and key learnings from the implementation of the following two systems implemented at a large, tertiary care children’s hospital: (1) epilepsy surgical candidate identification (or epilepsy ID) in an ambulatory neurology clinic; and (2) an automated clinical trial eligibility screener (ACTES) for the real-time identification of patients for research studies in a pediatric emergency department. Results: The epilepsy ID system performed as well as board-certified neurologists in identifying surgical candidates (with a sensitivity of 71% and positive predictive value of 77%). The ACTES system decreased coordinator screening time by 12.9%. The success of each project was largely dependent upon the collaboration between machine learning experts, research and operational information technology professionals, longitudinal support from clinical providers, and institutional leadership. Conclusions: These projects showcase novel interactions between machine learning recommendations and providers during clinical care. Our deployment provides seamless, real-time integration of AI technology to provide decision support and improve patient care.
- 16 December 2022
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 10 (12) , e37833
Open AccessImplementation of Machine Learning Pipelines for Clinical Practice: Development and Validation Study
Background: Artificial intelligence (AI) technologies, such as machine learning and natural language processing, have the potential to provide new insights into complex health data. Although powerful, these algorithms rarely move from experimental studies to direct clinical care implementation. Objective: We aimed to describe the key components for successful development and integration of two AI technology–based research pipelines for clinical practice. Methods: We summarized the approach, results, and key learnings from the implementation of the following two systems implemented at a large, tertiary care children’s hospital: (1) epilepsy surgical candidate identification (or epilepsy ID) in an ambulatory neurology clinic; and (2) an automated clinical trial eligibility screener (ACTES) for the real-time identification of patients for research studies in a pediatric emergency department. Results: The epilepsy ID system performed as well as board-certified neurologists in identifying surgical candidates (with a sensitivity of 71% and positive predictive value of 77%). The ACTES system decreased coordinator screening time by 12.9%. The success of each project was largely dependent upon the collaboration between machine learning experts, research and operational information technology professionals, longitudinal support from clinical providers, and institutional leadership. Conclusions: These projects showcase novel interactions between machine learning recommendations and providers during clinical care. Our deployment provides seamless, real-time integration of AI technology to provide decision support and improve patient care.
- 8 December 2021
- Published by Wiley in Pediatric Transplantation
- Vol. 26 (3) , e14204
External validation and comparison of risk score models in pediatric heart transplants
Background: Pediatric heart transplant (PHT) patients have the highest waitlist mortality of solid organ transplants, yet more than 40% of viable hearts are unutilized. A tool for risk prediction could impact these outcomes. This study aimed to compare and validate the PHT risk score models (RSMs) in the literature.Methods: The literature was reviewed to identify RSMs published. The United Network for Organ Sharing (UNOS) registry was used to validate the published models identified in a pediatric cohort ( 0.6]. Choudhry (AUC = 0.69) and SRTR (AUC = 0.71) remained the best predicting RSMs even after adjustment.Conclusion: Although the RSMs by SRTR and Choudhry provided the best prediction for 1‐year mortality, none demonstrated a strong (AUC ≥ 0.8) concordance statistic. All published studies lacked advanced analytical approaches and were derived from an inherently limited dataset.
- 8 December 2021
- Published by Wiley in Pediatric Transplantation
- Vol. 26 (3) , e14204
External validation and comparison of risk score models in pediatric heart transplants
Background: Pediatric heart transplant (PHT) patients have the highest waitlist mortality of solid organ transplants, yet more than 40% of viable hearts are unutilized. A tool for risk prediction could impact these outcomes. This study aimed to compare and validate the PHT risk score models (RSMs) in the literature.Methods: The literature was reviewed to identify RSMs published. The United Network for Organ Sharing (UNOS) registry was used to validate the published models identified in a pediatric cohort ( 0.6]. Choudhry (AUC = 0.69) and SRTR (AUC = 0.71) remained the best predicting RSMs even after adjustment.Conclusion: Although the RSMs by SRTR and Choudhry provided the best prediction for 1‐year mortality, none demonstrated a strong (AUC ≥ 0.8) concordance statistic. All published studies lacked advanced analytical approaches and were derived from an inherently limited dataset.
- Open Access
- 10 September 2021
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 23 (9) , e26231
Understanding Pediatric Surgery Cancellation: Geospatial Analysis
Background: Day-of-surgery cancellation (DoSC) represents a substantial wastage of hospital resources and can cause significant inconvenience to patients and families. Cancellation is reported to impact between 2% and 20% of the 50 million procedures performed annually in American hospitals. Up to 85% of cancellations may be amenable to the modification of patients’ and families’ behaviors. However, the factors underlying DoSC and the barriers experienced by families are not well understood.Objective: This study aims to conduct a geospatial analysis of patient-specific variables from electronic health records (EHRs) of Cincinnati Children’s Hospital Medical Center (CCHMC) and of Texas Children’s Hospital (TCH), as well as linked socioeconomic factors measured at the census tract level, to understand potential underlying contributors to disparities in DoSC rates across neighborhoods.Methods: The study population included pediatric patients who underwent scheduled surgeries at CCHMC and TCH. A 5-year data set was extracted from the CCHMC EHR, and addresses were geocoded. An equivalent set of data >5.7 years was extracted from the TCH EHR. Case-based data related to patients’ health care use were aggregated at the census tract level. Community-level variables were extracted from the American Community Survey as surrogates for patients’ socioeconomic and minority status as well as markers of the surrounding context. Leveraging the selected variables, we built spatial models to understand the variation in DoSC rates across census tracts. The findings were compared to those of the nonspatial regression and deep learning models. Model performance was evaluated from the root mean squared error (RMSE) using nested 10-fold cross-validation. Feature importance was evaluated by computing the increment of the RMSE when a single variable was shuffled within the data set.Results: Data collection yielded sets of 463 census tracts at CCHMC (DoSC rates 1.2%-12.5%) and 1024 census tracts at TCH (DoSC rates 3%-12.2%). For CCHMC, an L2-normalized generalized linear regression model achieved the best performance in predicting all-cause DoSC rate (RMSE 1.299%, 95% CI 1.21%-1.387%); however, its improvement over others was marginal. For TCH, an L2-normalized generalized linear regression model also performed best (RMSE 1.305%, 95% CI 1.257%-1.352%). All-cause DoSC rate at CCHMC was predicted most strongly by previous no show. As for community-level data, the proportion of African American inhabitants per census tract was consistently an important predictor. In the Texas area, the proportion of overcrowded households was salient to DoSC rate.Conclusions: Our findings suggest that geospatial analysis offers potential for use in targeting interventions for census tracts at a higher risk of cancellation. Our study also demonstrates the importance of home location, socioeconomic disadvantage, and racial minority status on the DoSC of children’s surgery. The success of future efforts to reduce cancellation may benefit from taking social, economic, and cultural issues into account.
- 10 September 2021
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 23 (9) , e26231
Open AccessUnderstanding Pediatric Surgery Cancellation: Geospatial Analysis
Background: Day-of-surgery cancellation (DoSC) represents a substantial wastage of hospital resources and can cause significant inconvenience to patients and families. Cancellation is reported to impact between 2% and 20% of the 50 million procedures performed annually in American hospitals. Up to 85% of cancellations may be amenable to the modification of patients’ and families’ behaviors. However, the factors underlying DoSC and the barriers experienced by families are not well understood.Objective: This study aims to conduct a geospatial analysis of patient-specific variables from electronic health records (EHRs) of Cincinnati Children’s Hospital Medical Center (CCHMC) and of Texas Children’s Hospital (TCH), as well as linked socioeconomic factors measured at the census tract level, to understand potential underlying contributors to disparities in DoSC rates across neighborhoods.Methods: The study population included pediatric patients who underwent scheduled surgeries at CCHMC and TCH. A 5-year data set was extracted from the CCHMC EHR, and addresses were geocoded. An equivalent set of data >5.7 years was extracted from the TCH EHR. Case-based data related to patients’ health care use were aggregated at the census tract level. Community-level variables were extracted from the American Community Survey as surrogates for patients’ socioeconomic and minority status as well as markers of the surrounding context. Leveraging the selected variables, we built spatial models to understand the variation in DoSC rates across census tracts. The findings were compared to those of the nonspatial regression and deep learning models. Model performance was evaluated from the root mean squared error (RMSE) using nested 10-fold cross-validation. Feature importance was evaluated by computing the increment of the RMSE when a single variable was shuffled within the data set.Results: Data collection yielded sets of 463 census tracts at CCHMC (DoSC rates 1.2%-12.5%) and 1024 census tracts at TCH (DoSC rates 3%-12.2%). For CCHMC, an L2-normalized generalized linear regression model achieved the best performance in predicting all-cause DoSC rate (RMSE 1.299%, 95% CI 1.21%-1.387%); however, its improvement over others was marginal. For TCH, an L2-normalized generalized linear regression model also performed best (RMSE 1.305%, 95% CI 1.257%-1.352%). All-cause DoSC rate at CCHMC was predicted most strongly by previous no show. As for community-level data, the proportion of African American inhabitants per census tract was consistently an important predictor. In the Texas area, the proportion of overcrowded households was salient to DoSC rate.Conclusions: Our findings suggest that geospatial analysis offers potential for use in targeting interventions for census tracts at a higher risk of cancellation. Our study also demonstrates the importance of home location, socioeconomic disadvantage, and racial minority status on the DoSC of children’s surgery. The success of future efforts to reduce cancellation may benefit from taking social, economic, and cultural issues into account.
- Open Access
- 1 August 2021
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 28 (10) , 2116-2127
Automated detection of substance use information from electronic health records for a pediatric population
Objective: Substance use screening in adolescence is unstandardized and often documented in clinical notes, rather than in structured electronic health records (EHRs). The objective of this study was to integrate logic rules with state-of-the-art natural language processing (NLP) and machine learning technologies to detect substance use information from both structured and unstructured EHR data. Materials and Methods: Pediatric patients (10-20 years of age) with any encounter between July 1, 2012, and October 31, 2017, were included (n = 3890 patients; 19 478 encounters). EHR data were extracted at each encounter, manually reviewed for substance use (alcohol, tobacco, marijuana, opiate, any use), and coded as lifetime use, current use, or family use. Logic rules mapped structured EHR indicators to screening results. A knowledge-based NLP system and a deep learning model detected substance use information from unstructured clinical narratives. System performance was evaluated using positive predictive value, sensitivity, negative predictive value, specificity, and area under the receiver-operating characteristic curve (AUC). Results: The dataset included 17 235 structured indicators and 27 141 clinical narratives. Manual review of clinical narratives captured 94.0% of positive screening results, while structured EHR data captured 22.0%. Logic rules detected screening results from structured data with 1.0 and 0.99 for sensitivity and specificity, respectively. The knowledge-based system detected substance use information from clinical narratives with 0.86, 0.79, and 0.88 for AUC, sensitivity, and specificity, respectively. The deep learning model further improved detection capacity, achieving 0.88, 0.81, and 0.85 for AUC, sensitivity, and specificity, respectively. Finally, integrating predictions from structured and unstructured data achieved high detection capacity across all cases (0.96, 0.85, and 0.87 for AUC, sensitivity, and specificity, respectively). Conclusions: It is feasible to detect substance use screening and results among pediatric patients using logic rules, NLP, and machine learning technologies.
- 1 August 2021
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 28 (10) , 2116-2127
Open AccessAutomated detection of substance use information from electronic health records for a pediatric population
Objective: Substance use screening in adolescence is unstandardized and often documented in clinical notes, rather than in structured electronic health records (EHRs). The objective of this study was to integrate logic rules with state-of-the-art natural language processing (NLP) and machine learning technologies to detect substance use information from both structured and unstructured EHR data. Materials and Methods: Pediatric patients (10-20 years of age) with any encounter between July 1, 2012, and October 31, 2017, were included (n = 3890 patients; 19 478 encounters). EHR data were extracted at each encounter, manually reviewed for substance use (alcohol, tobacco, marijuana, opiate, any use), and coded as lifetime use, current use, or family use. Logic rules mapped structured EHR indicators to screening results. A knowledge-based NLP system and a deep learning model detected substance use information from unstructured clinical narratives. System performance was evaluated using positive predictive value, sensitivity, negative predictive value, specificity, and area under the receiver-operating characteristic curve (AUC). Results: The dataset included 17 235 structured indicators and 27 141 clinical narratives. Manual review of clinical narratives captured 94.0% of positive screening results, while structured EHR data captured 22.0%. Logic rules detected screening results from structured data with 1.0 and 0.99 for sensitivity and specificity, respectively. The knowledge-based system detected substance use information from clinical narratives with 0.86, 0.79, and 0.88 for AUC, sensitivity, and specificity, respectively. The deep learning model further improved detection capacity, achieving 0.88, 0.81, and 0.85 for AUC, sensitivity, and specificity, respectively. Finally, integrating predictions from structured and unstructured data achieved high detection capacity across all cases (0.96, 0.85, and 0.87 for AUC, sensitivity, and specificity, respectively). Conclusions: It is feasible to detect substance use screening and results among pediatric patients using logic rules, NLP, and machine learning technologies.
- Open Access
- 4 December 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Understanding Pediatric Surgery Cancellation: Geospatial Analysis (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Day-of-surgery cancellation (DoSC) represents a substantial wastage of hospital resources and can cause significant inconvenience to patients and families. Cancellation is reported to impact between 2% and 20% of the 50 million procedures performed annually in American hospitals. Up to 85% of cancellations may be amenable to the modification of patients’ and families’ behaviors. However, the factors underlying DoSC and the barriers experienced by families are not well understood. Objective: This study aims to conduct a geospatial analysis of patient-specific variables from electronic health records (EHRs) of Cincinnati Children’s Hospital Medical Center (CCHMC) and of Texas Children’s Hospital (TCH), as well as linked socioeconomic factors measured at the census tract level, to understand potential underlying contributors to disparities in DoSC rates across neighborhoods. Methods: The study population included pediatric patients who underwent scheduled surgeries at CCHMC and TCH. A 5-year data set was extracted from the CCHMC EHR, and addresses were geocoded. An equivalent set of data >5.7 years was extracted from the TCH EHR. Case-based data related to patients’ health care use were aggregated at the census tract level. Community-level variables were extracted from the American Community Survey as surrogates for patients’ socioeconomic and minority status as well as markers of the surrounding context. Leveraging the selected variables, we built spatial models to understand the variation in DoSC rates across census tracts. The findings were compared to those of the nonspatial regression and deep learning models. Model performance was evaluated from the root mean squared error (RMSE) using nested 10-fold cross-validation. Feature importance was evaluated by computing the increment of the RMSE when a single variable was shuffled within the data set. Results: Data collection yielded sets of 463 census tracts at CCHMC (DoSC rates 1.2%-12.5%) and 1024 census tracts at TCH (DoSC rates 3%-12.2%). For CCHMC, an L2-normalized generalized linear regression model achieved the best performance in predicting all-cause DoSC rate (RMSE 1.299%, 95% CI 1.21%-1.387%); however, its improvement over others was marginal. For TCH, an L2-normalized generalized linear regression model also performed best (RMSE 1.305%, 95% CI 1.257%-1.352%). All-cause DoSC rate at CCHMC was predicted most strongly by previous no show. As for community-level data, the proportion of African American inhabitants per census tract was consistently an important predictor. In the Texas area, the proportion of overcrowded households was salient to DoSC rate. Conclusions: Our findings suggest that geospatial analysis offers potential for use in targeting interventions for census tracts at a higher risk of cancellation. Our study also demonstrates the importance of home location, socioeconomic disadvantage, and racial minority status on the DoSC of children’s surgery. The success of future efforts to reduce cancellation may benefit from taking social, economic, and cultural issues into account.
- 4 December 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessUnderstanding Pediatric Surgery Cancellation: Geospatial Analysis (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Day-of-surgery cancellation (DoSC) represents a substantial wastage of hospital resources and can cause significant inconvenience to patients and families. Cancellation is reported to impact between 2% and 20% of the 50 million procedures performed annually in American hospitals. Up to 85% of cancellations may be amenable to the modification of patients’ and families’ behaviors. However, the factors underlying DoSC and the barriers experienced by families are not well understood. Objective: This study aims to conduct a geospatial analysis of patient-specific variables from electronic health records (EHRs) of Cincinnati Children’s Hospital Medical Center (CCHMC) and of Texas Children’s Hospital (TCH), as well as linked socioeconomic factors measured at the census tract level, to understand potential underlying contributors to disparities in DoSC rates across neighborhoods. Methods: The study population included pediatric patients who underwent scheduled surgeries at CCHMC and TCH. A 5-year data set was extracted from the CCHMC EHR, and addresses were geocoded. An equivalent set of data >5.7 years was extracted from the TCH EHR. Case-based data related to patients’ health care use were aggregated at the census tract level. Community-level variables were extracted from the American Community Survey as surrogates for patients’ socioeconomic and minority status as well as markers of the surrounding context. Leveraging the selected variables, we built spatial models to understand the variation in DoSC rates across census tracts. The findings were compared to those of the nonspatial regression and deep learning models. Model performance was evaluated from the root mean squared error (RMSE) using nested 10-fold cross-validation. Feature importance was evaluated by computing the increment of the RMSE when a single variable was shuffled within the data set. Results: Data collection yielded sets of 463 census tracts at CCHMC (DoSC rates 1.2%-12.5%) and 1024 census tracts at TCH (DoSC rates 3%-12.2%). For CCHMC, an L2-normalized generalized linear regression model achieved the best performance in predicting all-cause DoSC rate (RMSE 1.299%, 95% CI 1.21%-1.387%); however, its improvement over others was marginal. For TCH, an L2-normalized generalized linear regression model also performed best (RMSE 1.305%, 95% CI 1.257%-1.352%). All-cause DoSC rate at CCHMC was predicted most strongly by previous no show. As for community-level data, the proportion of African American inhabitants per census tract was consistently an important predictor. In the Texas area, the proportion of overcrowded households was salient to DoSC rate. Conclusions: Our findings suggest that geospatial analysis offers potential for use in targeting interventions for census tracts at a higher risk of cancellation. Our study also demonstrates the importance of home location, socioeconomic disadvantage, and racial minority status on the DoSC of children’s surgery. The success of future efforts to reduce cancellation may benefit from taking social, economic, and cultural issues into account.
- Open Access
- 2 December 2020
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 8 (12) , e22031
The Generalizability of a Medication Administration Discrepancy Detection System: Quantitative Comparative Analysis
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: As a result of the overwhelming proportion of medication errors occurring each year, there has been an increased focus on developing medication error prevention strategies. Recent advances in electronic health record (EHR) technologies allows institutions the opportunity to identify medication administration error (MAE) events in real-time through computerized algorithms. MED.Safe, a software package comprised of medication discrepancy detection algorithms, was developed to meet this need by performing an automated comparison of medication orders to administration records. In order to demonstrate generalizability in other care settings, software such as this must be tested and validated in settings distinct from the development site. Objective: To determine the portability and generalizability of the MED.Safe software at a second site by assessing the performance and fit of the algorithms through comparison of discrepancy rates and other metrics across institutions. Methods: The MED.Safe software package was executed on medication use data from the implementation site to generate prescribing ratios and discrepancy rates. A retrospective analysis of medication prescribing and documentation patterns was then performed on the results and compared to those from the development site to determine the algorithmic performance and fit. Variance in performance from the development site was further explored and characterized. Results: Compared to the development site, the implementation site had lower Audit/order ratios and higher MAR/(Order+Audit) ratios. The discrepancy rates on the implementation site were consistently higher than those from the development site. Three drivers for the higher discrepancy rates included alternative clinical workflow using orders with dosing ranges, a data extract, transfer, and load issue causing modified order data to overwrite original order values in the EHRs, and delayed EHR documentation of verbal orders. Opportunities for improvement were identified and applied using a software update, which decreased false positive discrepancies and improved overall fit. Conclusions: The execution of MED.Safe at a second site was feasible and effective in the detection of medication administration discrepancies. A comparison of medication ordering, administration, and discrepancy rates identified areas where MED.Safe could be improved through customization. One modification of MED.Safe through deployment of a software update improved the overall algorithmic fit at the implementation site. More flexible customizations to accommodate different clinical practice patterns could improve MED.Safe’s fit at new sites.
- 2 December 2020
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 8 (12) , e22031
Open AccessThe Generalizability of a Medication Administration Discrepancy Detection System: Quantitative Comparative Analysis
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: As a result of the overwhelming proportion of medication errors occurring each year, there has been an increased focus on developing medication error prevention strategies. Recent advances in electronic health record (EHR) technologies allows institutions the opportunity to identify medication administration error (MAE) events in real-time through computerized algorithms. MED.Safe, a software package comprised of medication discrepancy detection algorithms, was developed to meet this need by performing an automated comparison of medication orders to administration records. In order to demonstrate generalizability in other care settings, software such as this must be tested and validated in settings distinct from the development site. Objective: To determine the portability and generalizability of the MED.Safe software at a second site by assessing the performance and fit of the algorithms through comparison of discrepancy rates and other metrics across institutions. Methods: The MED.Safe software package was executed on medication use data from the implementation site to generate prescribing ratios and discrepancy rates. A retrospective analysis of medication prescribing and documentation patterns was then performed on the results and compared to those from the development site to determine the algorithmic performance and fit. Variance in performance from the development site was further explored and characterized. Results: Compared to the development site, the implementation site had lower Audit/order ratios and higher MAR/(Order+Audit) ratios. The discrepancy rates on the implementation site were consistently higher than those from the development site. Three drivers for the higher discrepancy rates included alternative clinical workflow using orders with dosing ranges, a data extract, transfer, and load issue causing modified order data to overwrite original order values in the EHRs, and delayed EHR documentation of verbal orders. Opportunities for improvement were identified and applied using a software update, which decreased false positive discrepancies and improved overall fit. Conclusions: The execution of MED.Safe at a second site was feasible and effective in the detection of medication administration discrepancies. A comparison of medication ordering, administration, and discrepancy rates identified areas where MED.Safe could be improved through customization. One modification of MED.Safe through deployment of a software update improved the overall algorithmic fit at the implementation site. More flexible customizations to accommodate different clinical practice patterns could improve MED.Safe’s fit at new sites.
- Open Access
- 2 September 2020
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 8 (9) , e19774
Integrating and Evaluating the Data Quality and Utility of Smart Pump Information in Detecting Medication Administration Errors: Evaluation Study
Background: At present, electronic health records (EHRs) are the central focus of clinical informatics given their role as the primary source of clinical data. Despite their granularity, the EHR data heavily rely on manual input and are prone to human errors. Many other sources of data exist in the clinical setting, including digital medical devices such as smart infusion pumps. When incorporated with prescribing data from EHRs, smart pump records (SPRs) are capable of shedding light on actions that take place during the medication use process. However, harmoniz-ing the 2 sources is hindered by multiple technical challenges, and the data quality and utility of SPRs have not been fully realized. Objective: This study aims to evaluate the quality and utility of SPRs incorporated with EHR data in detecting medication administration errors. Our overarching hypothesis is that SPRs would contribute unique information in the med-ication use process, enabling more comprehensive detection of discrepancies and potential errors in medication administration. Methods: We evaluated the medication use process of 9 high-risk medications for patients admitted to the neonatal inten-sive care unit during a 1-year period. An automated algorithm was developed to align SPRs with their medica-tion orders in the EHRs using patient ID, medication name, and timestamp. The aligned data were manually re-viewed by a clinical research coordinator and 2 pediatric physicians to identify discrepancies in medication ad-ministration. The data quality of SPRs was assessed with the proportion of information that was linked to valid EHR orders. To evaluate their utility, we compared the frequency and severity of discrepancies captured by the SPR and EHR data, respectively. A novel concordance assessment was also developed to understand the detec-tion power and capabilities of SPR and EHR data. Results: Approximately 70% of the SPRs contained valid patient IDs and medication names, making them feasible for data integration. After combining the 2 sources, the investigative team reviewed 2307 medication orders with 10,575 medication administration records (MARs) and 23,397 SPRs. A total of 321 MAR and 682 SPR dis-crepancies were identified, with vasopressors showing the highest discrepancy rates, followed by narcotics and total parenteral nutrition. Compared with EHR MARs, substantial dosing discrepancies were more commonly detectable using the SPRs. The concordance analysis showed little overlap between MAR and SPR discrepan-cies, with most discrepancies captured by the SPR data. Conclusions: We integrated smart infusion pump information with EHR data to analyze the most error-prone phases of the medication lifecycle. The findings suggested that SPRs could be a more reliable data source for medication error detection. Ultimately, it is imperative to integrate SPR information with EHR data to fully detect and mitigate medication administration errors in the clinical setting.
- 2 September 2020
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 8 (9) , e19774
Open AccessIntegrating and Evaluating the Data Quality and Utility of Smart Pump Information in Detecting Medication Administration Errors: Evaluation Study
Background: At present, electronic health records (EHRs) are the central focus of clinical informatics given their role as the primary source of clinical data. Despite their granularity, the EHR data heavily rely on manual input and are prone to human errors. Many other sources of data exist in the clinical setting, including digital medical devices such as smart infusion pumps. When incorporated with prescribing data from EHRs, smart pump records (SPRs) are capable of shedding light on actions that take place during the medication use process. However, harmoniz-ing the 2 sources is hindered by multiple technical challenges, and the data quality and utility of SPRs have not been fully realized. Objective: This study aims to evaluate the quality and utility of SPRs incorporated with EHR data in detecting medication administration errors. Our overarching hypothesis is that SPRs would contribute unique information in the med-ication use process, enabling more comprehensive detection of discrepancies and potential errors in medication administration. Methods: We evaluated the medication use process of 9 high-risk medications for patients admitted to the neonatal inten-sive care unit during a 1-year period. An automated algorithm was developed to align SPRs with their medica-tion orders in the EHRs using patient ID, medication name, and timestamp. The aligned data were manually re-viewed by a clinical research coordinator and 2 pediatric physicians to identify discrepancies in medication ad-ministration. The data quality of SPRs was assessed with the proportion of information that was linked to valid EHR orders. To evaluate their utility, we compared the frequency and severity of discrepancies captured by the SPR and EHR data, respectively. A novel concordance assessment was also developed to understand the detec-tion power and capabilities of SPR and EHR data. Results: Approximately 70% of the SPRs contained valid patient IDs and medication names, making them feasible for data integration. After combining the 2 sources, the investigative team reviewed 2307 medication orders with 10,575 medication administration records (MARs) and 23,397 SPRs. A total of 321 MAR and 682 SPR dis-crepancies were identified, with vasopressors showing the highest discrepancy rates, followed by narcotics and total parenteral nutrition. Compared with EHR MARs, substantial dosing discrepancies were more commonly detectable using the SPRs. The concordance analysis showed little overlap between MAR and SPR discrepan-cies, with most discrepancies captured by the SPR data. Conclusions: We integrated smart infusion pump information with EHR data to analyze the most error-prone phases of the medication lifecycle. The findings suggested that SPRs could be a more reliable data source for medication error detection. Ultimately, it is imperative to integrate SPR information with EHR data to fully detect and mitigate medication administration errors in the clinical setting.
- Open Access
- 9 July 2020
- Published by JMIR Publications Inc. in JMIR Preprints
The Generalizability of a Medication Administration Discrepancy Detection System: Quantitative Comparative Analysis (Preprint)
BACKGROUND As a result of the overwhelming proportion of medication errors occurring each year, there has been an increased focus on developing medication error prevention strategies. Recent advances in electronic health record (EHR) technologies allow institutions the opportunity to identify medication administration error events in real time through computerized algorithms. MED.Safe, a software package comprising medication discrepancy detection algorithms, was developed to meet this need by performing an automated comparison of medication orders to medication administration records (MARs). In order to demonstrate generalizability in other care settings, software such as this must be tested and validated in settings distinct from the development site. OBJECTIVE The purpose of this study is to determine the portability and generalizability of the MED.Safe software at a second site by assessing the performance and fit of the algorithms through comparison of discrepancy rates and other metrics across institutions. METHODS The MED.Safe software package was executed on medication use data from the implementation site to generate prescribing ratios and discrepancy rates. A retrospective analysis of medication prescribing and documentation patterns was then performed on the results and compared to those from the development site to determine the algorithmic performance and fit. Variance in performance from the development site was further explored and characterized. RESULTS Compared to the development site, the implementation site had lower audit/order ratios and higher MAR/(order + audit) ratios. The discrepancy rates on the implementation site were consistently higher than those from the development site. Three drivers for the higher discrepancy rates were alternative clinical workflow using orders with dosing ranges; a data extract, transfer, and load issue causing modified order data to overwrite original order values in the EHRs; and delayed EHR documentation of verbal orders. Opportunities for improvement were identified and applied using a software update, which decreased false-positive discrepancies and improved overall fit. CONCLUSIONS The execution of MED.Safe at a second site was feasible and effective in the detection of medication administration discrepancies. A comparison of medication ordering, administration, and discrepancy rates identified areas where MED.Safe could be improved through customization. One modification of MED.Safe through deployment of a software update improved the overall algorithmic fit at the implementation site. More flexible customizations to accommodate different clinical practice patterns could improve MED.Safe’s fit at new sites.
- 9 July 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessThe Generalizability of a Medication Administration Discrepancy Detection System: Quantitative Comparative Analysis (Preprint)
BACKGROUND As a result of the overwhelming proportion of medication errors occurring each year, there has been an increased focus on developing medication error prevention strategies. Recent advances in electronic health record (EHR) technologies allow institutions the opportunity to identify medication administration error events in real time through computerized algorithms. MED.Safe, a software package comprising medication discrepancy detection algorithms, was developed to meet this need by performing an automated comparison of medication orders to medication administration records (MARs). In order to demonstrate generalizability in other care settings, software such as this must be tested and validated in settings distinct from the development site. OBJECTIVE The purpose of this study is to determine the portability and generalizability of the MED.Safe software at a second site by assessing the performance and fit of the algorithms through comparison of discrepancy rates and other metrics across institutions. METHODS The MED.Safe software package was executed on medication use data from the implementation site to generate prescribing ratios and discrepancy rates. A retrospective analysis of medication prescribing and documentation patterns was then performed on the results and compared to those from the development site to determine the algorithmic performance and fit. Variance in performance from the development site was further explored and characterized. RESULTS Compared to the development site, the implementation site had lower audit/order ratios and higher MAR/(order + audit) ratios. The discrepancy rates on the implementation site were consistently higher than those from the development site. Three drivers for the higher discrepancy rates were alternative clinical workflow using orders with dosing ranges; a data extract, transfer, and load issue causing modified order data to overwrite original order values in the EHRs; and delayed EHR documentation of verbal orders. Opportunities for improvement were identified and applied using a software update, which decreased false-positive discrepancies and improved overall fit. CONCLUSIONS The execution of MED.Safe at a second site was feasible and effective in the detection of medication administration discrepancies. A comparison of medication ordering, administration, and discrepancy rates identified areas where MED.Safe could be improved through customization. One modification of MED.Safe through deployment of a software update improved the overall algorithmic fit at the implementation site. More flexible customizations to accommodate different clinical practice patterns could improve MED.Safe’s fit at new sites.
- Open Access
- 1 May 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Integrating and Evaluating the Data Quality and Utility of Smart Pump Information in Detecting Medication Administration Errors: Evaluation Study (Preprint)
BACKGROUND At present, electronic health records (EHRs) are the central focus of clinical informatics given their role as the primary source of clinical data. Despite their granularity, the EHR data heavily rely on manual input and are prone to human errors. Many other sources of data exist in the clinical setting, including digital medical devices such as smart infusion pumps. When incorporated with prescribing data from EHRs, smart pump records (SPRs) are capable of shedding light on actions that take place during the medication use process. However, harmoniz-ing the 2 sources is hindered by multiple technical challenges, and the data quality and utility of SPRs have not been fully realized. OBJECTIVE This study aims to evaluate the quality and utility of SPRs incorporated with EHR data in detecting medication administration errors. Our overarching hypothesis is that SPRs would contribute unique information in the med-ication use process, enabling more comprehensive detection of discrepancies and potential errors in medication administration. METHODS We evaluated the medication use process of 9 high-risk medications for patients admitted to the neonatal inten-sive care unit during a 1-year period. An automated algorithm was developed to align SPRs with their medica-tion orders in the EHRs using patient ID, medication name, and timestamp. The aligned data were manually re-viewed by a clinical research coordinator and 2 pediatric physicians to identify discrepancies in medication ad-ministration. The data quality of SPRs was assessed with the proportion of information that was linked to valid EHR orders. To evaluate their utility, we compared the frequency and severity of discrepancies captured by the SPR and EHR data, respectively. A novel concordance assessment was also developed to understand the detec-tion power and capabilities of SPR and EHR data. RESULTS Approximately 70% of the SPRs contained valid patient IDs and medication names, making them feasible for data integration. After combining the 2 sources, the investigative team reviewed 2307 medication orders with 10,575 medication administration records (MARs) and 23,397 SPRs. A total of 321 MAR and 682 SPR dis-crepancies were identified, with vasopressors showing the highest discrepancy rates, followed by narcotics and total parenteral nutrition. Compared with EHR MARs, substantial dosing discrepancies were more commonly detectable using the SPRs. The concordance analysis showed little overlap between MAR and SPR discrepan-cies, with most discrepancies captured by the SPR data. CONCLUSIONS We integrated smart infusion pump information with EHR data to analyze the most error-prone phases of the medication lifecycle. The findings suggested that SPRs could be a more reliable data source for medication error detection. Ultimately, it is imperative to integrate SPR information with EHR data to fully detect and mitigate medication administration errors in the clinical setting.
- 1 May 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessIntegrating and Evaluating the Data Quality and Utility of Smart Pump Information in Detecting Medication Administration Errors: Evaluation Study (Preprint)
BACKGROUND At present, electronic health records (EHRs) are the central focus of clinical informatics given their role as the primary source of clinical data. Despite their granularity, the EHR data heavily rely on manual input and are prone to human errors. Many other sources of data exist in the clinical setting, including digital medical devices such as smart infusion pumps. When incorporated with prescribing data from EHRs, smart pump records (SPRs) are capable of shedding light on actions that take place during the medication use process. However, harmoniz-ing the 2 sources is hindered by multiple technical challenges, and the data quality and utility of SPRs have not been fully realized. OBJECTIVE This study aims to evaluate the quality and utility of SPRs incorporated with EHR data in detecting medication administration errors. Our overarching hypothesis is that SPRs would contribute unique information in the med-ication use process, enabling more comprehensive detection of discrepancies and potential errors in medication administration. METHODS We evaluated the medication use process of 9 high-risk medications for patients admitted to the neonatal inten-sive care unit during a 1-year period. An automated algorithm was developed to align SPRs with their medica-tion orders in the EHRs using patient ID, medication name, and timestamp. The aligned data were manually re-viewed by a clinical research coordinator and 2 pediatric physicians to identify discrepancies in medication ad-ministration. The data quality of SPRs was assessed with the proportion of information that was linked to valid EHR orders. To evaluate their utility, we compared the frequency and severity of discrepancies captured by the SPR and EHR data, respectively. A novel concordance assessment was also developed to understand the detec-tion power and capabilities of SPR and EHR data. RESULTS Approximately 70% of the SPRs contained valid patient IDs and medication names, making them feasible for data integration. After combining the 2 sources, the investigative team reviewed 2307 medication orders with 10,575 medication administration records (MARs) and 23,397 SPRs. A total of 321 MAR and 682 SPR dis-crepancies were identified, with vasopressors showing the highest discrepancy rates, followed by narcotics and total parenteral nutrition. Compared with EHR MARs, substantial dosing discrepancies were more commonly detectable using the SPRs. The concordance analysis showed little overlap between MAR and SPR discrepan-cies, with most discrepancies captured by the SPR data. CONCLUSIONS We integrated smart infusion pump information with EHR data to analyze the most error-prone phases of the medication lifecycle. The findings suggested that SPRs could be a more reliable data source for medication error detection. Ultimately, it is imperative to integrate SPR information with EHR data to fully detect and mitigate medication administration errors in the clinical setting.
- 26 April 2020
- Published by Taylor & Francis in Expert Review of Respiratory Medicine
- Vol. 14 (7) , 737-748
Influences of environmental exposures on individuals living with cystic fibrosis
Natural, social, and constructed environments play a critical role in the development and exacerbation of respiratory diseases. However, less is known regarding the influence of these environmental...
- 26 April 2020
- Published by Taylor & Francis in Expert Review of Respiratory Medicine
- Vol. 14 (7) , 737-748
Influences of environmental exposures on individuals living with cystic fibrosis
Natural, social, and constructed environments play a critical role in the development and exacerbation of respiratory diseases. However, less is known regarding the influence of these environmental...