SSSteven Simon

Steven Simon

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Steven Simon has a total of 65 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Impact of an Alert-Based Inpatient Clinical Decision Support Tool to Prevent Drug-Induced Long QT Syndrome: Large-Scale, System-Wide Observational Study

    by Katy E Trinkley, Steven T Simon, Michael A Rosenberg

    Background: Prevention of drug-induced QT prolongation (diLQTS) has been the focus of many system-wide clinical decision support (CDS) tools, which can be directly embedded within the framework of the electronic health record system and triggered to alert in high-risk patients when a known QT-prolonging medication is ordered. Justification for these CDS systems typically lies in the ability to accurately predict which patients are at high risk; however, it is not always evident that identification of risk alone is sufficient for appropriate CDS implementation. Objective: In this investigation, we examined the impact of a system-wide, alert-based, inpatient CDS tool designed to prevent diLQTS across 10 known QT-prolonging medications. Methods: We compared the risk of diLQTS, duration of hospitalization, and in- and out-of-hospital mortality before and after implementation of the CDS system in 178,097 hospitalizations among 102,847 patients. We also compared outcomes between those in whom an alert fired and those in whom it did not, and within the various responses to the alert by providers. Analyses were adjusted for age, sex, race and ethnicity, inpatient location, electrolyte values, and comorbidities, with the latter processed using an unsupervised clustering analysis applied to the top 500 most common medications and diagnosis codes, respectively. Results: We found that the simple, rule-based logic of the CDS (any prior electrocardiograph with heart rate–corrected QT interval (QTc)≥500 ms) successfully identified patients at high risk of diLQTS with an odds ratio of 2.28 (95% CI 2.10-2.47, P<.001) among those in whom it fired. However, we did not identify any impact on the risk of diLQTS based on provider responses or on the risk of inpatient, 3-month, 6-month, or 1-year mortality. When compared with rates prior to implementation, the risk of diLQTS was not significantly different after the CDS tools were deployed across the system, although mortality was significantly higher after the tools were implemented. Conclusions: We found that despite successful identification of high-risk patients for diLQTS, deployment of an alert-based CDS did not impact the risk of diLQTS. These findings suggest that quantification of high risk may be insufficient rationale for implementation of a CDS system and that hospital systems should consider evaluation of the system in its entirety prior to adoption to improve clinical outcomes.


  • Open Access

    Impact of an Alert-Based Inpatient Clinical Decision Support Tool to Prevent Drug-Induced Long QT Syndrome: Large-Scale, System-Wide Observational Study (Preprint)

    by Katy E Trinkley, Steven T Simon, Michael A Rosenberg

    BACKGROUND Prevention of drug-induced QT prolongation (diLQTS) has been the focus of many system-wide clinical decision support (CDS) tools, which can be directly embedded within the framework of the electronic health record system and triggered to alert in high-risk patients when a known QT-prolonging medication is ordered. Justification for these CDS systems typically lies in the ability to accurately predict which patients are at high risk; however, it is not always evident that identification of risk alone is sufficient for appropriate CDS implementation. OBJECTIVE In this investigation, we examined the impact of a system-wide, alert-based, inpatient CDS tool designed to prevent diLQTS across 10 known QT-prolonging medications. METHODS We compared the risk of diLQTS, duration of hospitalization, and in- and out-of-hospital mortality before and after implementation of the CDS system in 178,097 hospitalizations among 102,847 patients. We also compared outcomes between those in whom an alert fired and those in whom it did not, and within the various responses to the alert by providers. Analyses were adjusted for age, sex, race and ethnicity, inpatient location, electrolyte values, and comorbidities, with the latter processed using an unsupervised clustering analysis applied to the top 500 most common medications and diagnosis codes, respectively. RESULTS We found that the simple, rule-based logic of the CDS (any prior electrocardiograph with heart rate–corrected QT interval (QTc)≥500 ms) successfully identified patients at high risk of diLQTS with an odds ratio of 2.28 (95% CI 2.10-2.47, P<.001) among those in whom it fired. However, we did not identify any impact on the risk of diLQTS based on provider responses or on the risk of inpatient, 3-month, 6-month, or 1-year mortality. When compared with rates prior to implementation, the risk of diLQTS was not significantly different after the CDS tools were deployed across the system, although mortality was significantly higher after the tools were implemented. CONCLUSIONS We found that despite successful identification of high-risk patients for diLQTS, deployment of an alert-based CDS did not impact the risk of diLQTS. These findings suggest that quantification of high risk may be insufficient rationale for implementation of a CDS system and that hospital systems should consider evaluation of the system in its entirety prior to adoption to improve clinical outcomes.

  • Open Access

    Interpretable Machine Learning Prediction of Drug-Induced QT Prolongation: Electronic Health Record Analysis

    by Steven T Simon, Katy E Trinkley, Daniel C Malone, Michael Aaron Rosenberg

    Background: Drug-induced long-QT syndrome (diLQTS) is a major concern among patients who are hospitalized, for whom prediction models capable of identifying individualized risk could be useful to guide monitoring. We have previously demonstrated the feasibility of machine learning to predict the risk of diLQTS, in which deep learning models provided superior accuracy for risk prediction, although these models were limited by a lack of interpretability. Objective: In this investigation, we sought to examine the potential trade-off between interpretability and predictive accuracy with the use of more complex models to identify patients at risk for diLQTS. We planned to compare a deep learning algorithm to predict diLQTS with a more interpretable algorithm based on cluster analysis that would allow medication- and subpopulation-specific evaluation of risk. Methods: We examined the risk of diLQTS among 35,639 inpatients treated between 2003 and 2018 with at least 1 of 39 medications associated with risk of diLQTS and who had an electrocardiogram in the system performed within 24 hours of medication administration. Predictors included over 22,000 diagnoses and medications at the time of medication administration, with cases of diLQTS defined as a corrected QT interval over 500 milliseconds after treatment with a culprit medication. The interpretable model was developed using cluster analysis (K=4 clusters), and risk was assessed for specific medications and classes of medications. The deep learning model was created using all predictors within a 6-layer neural network, based on previously identified hyperparameters. Results: Among the medications, we found that class III antiarrhythmic medications were associated with increased risk across all clusters, and that in patients who are noncritically ill without cardiovascular disease, propofol was associated with increased risk, whereas ondansetron was associated with decreased risk. Compared with deep learning, the interpretable approach was less accurate (area under the receiver operating characteristic curve: 0.65 vs 0.78), with comparable calibration. Conclusions: In summary, we found that an interpretable modeling approach was less accurate, but more clinically applicable, than deep learning for the prediction of diLQTS. Future investigations should consider this trade-off in the development of methods for clinical prediction.


  • Open Access

    The Impact of Time Horizon on Classification Accuracy: Application of Machine Learning to Prediction of Incident Coronary Heart Disease

    by Steven Simon, Divneet Mandair, Abdel Albakri, Alison Fohner, Noah Simon, Leslie Lange, Mary Biggs, Kenneth Mukamal, Bruce Psaty, Michael Rosenberg

    Background: Many machine learning approaches are limited to classification of outcomes rather than longitudinal prediction. One strategy to use machine learning in clinical risk prediction is to classify outcomes over a given time horizon. However, it is not well-known how to identify the optimal time horizon for risk prediction. Objective: In this study, we aim to identify an optimal time horizon for classification of incident myocardial infarction (MI) using machine learning approaches looped over outcomes with increasing time horizons. Additionally, we sought to compare the performance of these models with the traditional Framingham Heart Study (FHS) coronary heart disease gender-specific Cox proportional hazards regression model. Methods: We analyzed data from a single clinic visit of 5201 participants of a cardiovascular health study. We examined 61 variables collected from this baseline exam, including demographic and biologic data, medical history, medications, serum biomarkers, electrocardiographic, and echocardiographic data. We compared several machine learning methods (eg, random forest, L1 regression, gradient boosted decision tree, support vector machine, and k-nearest neighbor) trained to predict incident MI that occurred within time horizons ranging from 500-10,000 days of follow-up. Models were compared on a 20% held-out testing set using area under the receiver operating characteristic curve (AUROC). Variable importance was performed for random forest and L1 regression models across time points. We compared results with the FHS coronary heart disease gender-specific Cox proportional hazards regression functions. Results: There were 4190 participants included in the analysis, with 2522 (60.2%) female participants and an average age of 72.6 years. Over 10,000 days of follow-up, there were 813 incident MI events. The machine learning models were most predictive over moderate follow-up time horizons (ie, 1500-2500 days). Overall, the L1 (Lasso) logistic regression demonstrated the strongest classification accuracy across all time horizons. This model was most predictive at 1500 days follow-up, with an AUROC of 0.71. The most influential variables differed by follow-up time and model, with gender being the most important feature for the L1 regression and weight for the random forest model across all time frames. Compared with the Framingham Cox function, the L1 and random forest models performed better across all time frames beyond 1500 days. Conclusions: In a population free of coronary heart disease, machine learning techniques can be used to predict incident MI at varying time horizons with reasonable accuracy, with the strongest prediction accuracy in moderate follow-up periods. Validation across additional populations is needed to confirm the validity of this approach in risk prediction.


  • Open Access

    Qualitative Evaluation of an Artificial Intelligence–Based Clinical Decision Support System to Guide Rhythm Management of Atrial Fibrillation: Survey Study

    by John Stacy, Rachel Kim, Christopher Barrett, Balaviknesh Sekar, Steven Simon, Farnoush Banaei-Kashani, Michael A Rosenberg

    Background: Despite the numerous studies evaluating various rhythm control strategies for atrial fibrillation (AF), determination of the optimal strategy in a single patient is often based on trial and error, with no one-size-fits-all approach based on international guidelines/recommendations. The decision, therefore, remains personal and lends itself well to help from a clinical decision support system, specifically one guided by artificial intelligence (AI). QRhythm utilizes a 2-stage machine learning (ML) model to identify the optimal rhythm management strategy in a given patient based on a set of clinical factors, in which the model first uses supervised learning to predict the actions of an expert clinician and identifies the best strategy through reinforcement learning to obtain the best clinical outcome—a composite of symptomatic recurrence, hospitalization, and stroke. Objective: We qualitatively evaluated a novel, AI-based, clinical decision support system (CDSS) for AF rhythm management, called QRhythm, which uses both supervised and reinforcement learning to recommend either a rate control or one of 3 types of rhythm control strategies—external cardioversion, antiarrhythmic medication, or ablation—based on individual patient characteristics. Methods: Thirty-three clinicians, including cardiology attendings and fellows and internal medicine attendings and residents, performed an assessment of QRhythm, followed by a survey to assess relative comfort with automated CDSS in rhythm management and to examine areas for future development. Results: The 33 providers were surveyed with training levels ranging from resident to fellow to attending. Of the characteristics of the app surveyed, safety was most important to providers, with an average importance rating of 4.7 out of 5 (SD 0.72). This priority was followed by clinical integrity (a desire for the advice provided to make clinical sense; importance rating 4.5, SD 0.9), backward interpretability (transparency in the population used to create the algorithm; importance rating 4.3, SD 0.65), transparency of the algorithm (reasoning underlying the decisions made; importance rating 4.3, SD 0.88), and provider autonomy (the ability to challenge the decisions made by the model; importance rating 3.85, SD 0.83). Providers who used the app ranked the integrity of recommendations as their highest concern with ongoing clinical use of the model, followed by efficacy of the application and patient data security. Trust in the app varied; 1 (17%) provider responded that they somewhat disagreed with the statement, “I trust the recommendations provided by the QRhythm app,” 2 (33%) providers responded with neutrality to the statement, and 3 (50%) somewhat agreed with the statement. Conclusions: Safety of ML applications was the highest priority of the providers surveyed, and trust of such models remains varied. Widespread clinical acceptance of ML in health care is dependent on how much providers trust the algorithms. Building this trust involves ensuring transparency and interpretability of the model.


  • Open Access

    The Impact of Time Horizon on Classification Accuracy: Application of Machine Learning to Prediction of Incident Coronary Heart Disease (Preprint)

    by Steven Simon, Divneet Mandair, Abdel Albakri, Alison Fohner, Noah Simon, Leslie Lange, Mary Biggs, Ken Mukamal, Bruce Psaty, Michael Rosenberg

    BACKGROUND Many machine-learning (ML) approaches are limited to classification of outcomes, rather than longitudinal prediction. One strategy to use ML in clinical risk prediction is to classify outcomes over a given time horizon. However, it is not well-known how to identify the optimal time horizon for risk prediction. OBJECTIVE Here we aim to identify an optimal time horizon for classification of incident myocardial infarction using ML approaches looped over outcomes with increasing time horizons. METHODS We analyzed data from a single clinic visit of 5201 participants of the Cardiovascular Health Study. We examined 61 variables collected from this baseline exam including demographic and biologic data, medical history, medications, serum biomarkers, electrocardiographic, and echocardiographic data. We compared several machine learning methods (Random Forest, L1 Regression, Gradient Boosted Decision Tree, Support Vector Machines, and K-Nearest Neighbor) trained to predict incident MI that occurred within time horizons ranging from 500 through 10000 days of follow up. Models were compared on a 20% held-out testing set using area-under-receiver operator curve (AUC). Variable importance was performed for Random Forest and L1 Regression models across timepoints. We compared results with the Framingham coronary heart disease sex-specific Cox proportional hazards regression functions. RESULTS There were 4190 participants included in the analysis with 60.2% female and an average age of 72.6 years. Over the 10000 days of follow up, there were 813 incident myocardial infarction events. The ML models were most predictive over moderate follow up time horizons (1500-2500 days). Overall, the L1 (Lasso) logistic regression demonstrated the strongest classification accuracy across all time horizons. This model was most predictive at 1500 days follow up with an AUC of 0.71. The most influential variables differed by follow up time and model with gender being the most important feature for the L1 regression and weight for the random forest across all timeframes. Compared with the Framingham Cox function, the L1 and random forest models performed better across all timeframes beyond 1500 days. CONCLUSIONS In a population free of coronary heart disease, machine learning techniques can be utilized to predict incident myocardial infarction at varying time horizons with reasonable accuracy, with strongest prediction accuracy at moderate follow up periods. Validation across additional populations is needed to confirm a role for this approach in risk prediction.

  • Open Access

    Qualitative Evaluation of an Artificial Intelligence-based Clinical Decision Support System to Guide Rhythm Management of Atrial Fibrillation (Preprint)

    by John Stacy, Rachel Kim, Christopher D Barrett, Balaviknesh Sekar, Steve Simon, Farnoush Banaei-Kashani, Michael A. Rosenberg

    BACKGROUND Despite the numerous studies evaluating various rhythm-control strategies for patients with atrial fibrillation, determination of the optimal strategy in a single patient is often based on trial-and-error, with no one-size-fits-all approach based on international guideline recommendations. The decision, therefore, remains an individualized one that lends itself well to the aid of a clinical decision support system, specifically one guided by artificial intelligence (AI). OBJECTIVE In this study, we performed qualitative evaluation of a novel, AI-based, clinical decision support system (CDSS) for rhythm management of AF called QRhythm, which uses a combination of supervised and reinforcement learning to recommend either a rate-control or one of three types of rhythm-control strategies—external cardioversion, antiarrhythmic medication, or ablation—based on individual patient characteristics. METHODS QRhythm utilizes a two-stage machine learning model to identify the optimal rhythm-management strategy in a given patient based on a set of clinical factors, in which the model first uses supervised learning to mimic an expert clinician, and then applies reinforcement learning to identify the best strategy to obtain the best clinical outcome—a composite of symptomatic recurrence, hospitalization, and stroke. 33 clinical providers, including cardiologists and internal medicine physicians, performed an assessment of QRhythm, followed by a survey to assess relative comfort with automated CDSS in rhythm-management, and examine areas for future development. RESULTS 33 providers were surveyed with training levels ranging from resident to fellow to attending. Of the characteristics of the app surveyed, safety was most important to providers with an average importance rating of 4.7 out of 5 (SD 0.72). This priority was followed by a desire for the advice provided to make clinical sense with an importance of 4.5 out of 5 (SD 0.9), transparency in the population used to create the algorithm, 4.3 out of 5 (SD 0.65), and reasoning behind the decisions made, 4.3 out of 5 (SD0.88), and the ability to challenge the decisions made by the model, 3.85 out of 5 (SD 0.83). Providers that used the app ranked their highest concern with ongoing clinical use of the model as accuracy of recommendations followed by ineffectiveness of the application and patient data security. Trust in the app was varied. 16.7% of providers responded that they somewhat disagreed with the statement “I trust the recommendations provided by the QRhythm app,” 33.3% responded with neutrality to the statement, and 50% somewhat agreed with the statement. CONCLUSIONS Safety of machine learning applications was the number one priority of the providers surveyed, and trust of such models remains varied. Widespread clinical acceptance of machine learning in health care is dependent on how much providers trust the algorithms. Building this trust involves ensuring transparency and interpretability of the model.

  • Open Access

    Machine Learning Methodologies for Prediction of Rhythm-Control Strategy in Patients Diagnosed With Atrial Fibrillation: Observational, Retrospective, Case-Control Study

    by Rachel S Kim, Steven Simon, Brett Powers, Amneet Sandhu, Jose Sanchez, Ryan T Borne, Alexis Tumolo, Matthew Zipse, J Jason West, Ryan Aleong, et al.

    Background: The identification of an appropriate rhythm management strategy for patients diagnosed with atrial fibrillation (AF) remains a major challenge for providers. Although clinical trials have identified subgroups of patients in whom a rate- or rhythm-control strategy might be indicated to improve outcomes, the wide range of presentations and risk factors among patients presenting with AF makes such approaches challenging. The strength of electronic health records is the ability to build in logic to guide management decisions, such that the system can automatically identify patients in whom a rhythm-control strategy is more likely and can promote efficient referrals to specialists. However, like any clinical decision support tool, there is a balance between interpretability and accurate prediction. Objective: This study aims to create an electronic health record–based prediction tool to guide patient referral to specialists for rhythm-control management by comparing different machine learning algorithms. Methods: We compared machine learning models of increasing complexity and used up to 50,845 variables to predict the rhythm-control strategy in 42,022 patients within the University of Colorado Health system at the time of AF diagnosis. Models were evaluated on the basis of their classification accuracy, defined by the F1 score and other metrics, and interpretability, captured by inspection of the relative importance of each predictor. Results: We found that age was by far the strongest single predictor of a rhythm-control strategy but that greater accuracy could be achieved with more complex models incorporating neural networks and more predictors for each participant. We determined that the impact of better prediction models was notable primarily in the rate of inappropriate referrals for rhythm-control, in which more complex models provided an average of 20% fewer inappropriate referrals than simpler, more interpretable models. Conclusions: We conclude that any health care system seeking to incorporate algorithms to guide rhythm management for patients with AF will need to address this trade-off between prediction accuracy and model interpretability.


  • Open Access

    A Systems-Based Morbidity and Mortality Conference Was Associated With a Transient Reduction in ECG Completion Times

    by Andrew E. Md Levy, Melanie D. Whittington, Tyler J. Do Anstett, Steven T. Simon, Allen Med Wentworth, James E. Jr Carter, P. Michael Ho

    Background and Objectives: During its monthly morbidity and mortality conference (MMC), the University of Colorado Division of Cardiology reviewed a “near-miss” patient safety event involving the delayed completion of a Stat-priority (ie, statim, meaning high priority) electrocardiogram (ECG). Because critical and interprofessional stakeholders participated in the conference, we hypothesized that the MMC would be associated with reduced ECG completion times. Methods: Data were collected for in-hospital ECGs performed at the University of Colorado Hospital between January 1, 2017, and June 30, 2018. An interrupted time series analysis was used to estimate the immediate and ongoing impact of the MMC (held on February 28, 2018) on ECG completion times, stratified by order priority (Stat, Now, or Routine). The percentage of delayed Stat-priority ECGs was analyzed as a secondary outcome. Results: Before the MMC, ECG completion times were stable for all order priorities (P > .2), but the proportion of delayed Stat-priority ECGs increased from 5% in January 2017 to 20% in February 2018 (P < .01). The MMC was associated with an immediate reduction in average daily ECG completion times for Routine (−18.4 minutes, P = .03) and Now (−8 minutes, P = .024) priority ECGs. No reduction was seen for Stat ECGs (P = .97), though the percentage of delayed Stat ECGs stopped increasing (P = .63). In the post-MMC period, completion times for Routine-priority ECGs increased and approached pre-MMC levels. Conclusions: The MMC was associated with an immediate, but temporary, improvement in ECG completion times. Although the observed clinical benefit of the MMC is novel, these data support the need for more durable reforms to sustain initial improvements.