DRDiana Manuela Roccaro-Waldmeyer

Diana Manuela Roccaro-Waldmeyer

*This is an algorithmically generated author record based on publicly available information.

Diana Manuela Roccaro-Waldmeyer has a total of 27 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals

    by Patricia Cabanillas Silva, Hong Sun, Mohamed Rezk, Diana M Roccaro-Waldmeyer, Janis Fliegenschmidt, Nikolai Hulde, Vera von Dossow, Laurent Meesseman, Kristof Depraetere, Joerg Stieg, et al.

    Background: In recent years, machine learning (ML)–based models have been widely used in clinical domains to predict clinical risk events. However, in production, the performances of such models heavily rely on changes in the system and data. The dynamic nature of the system environment, characterized by continuous changes, has significant implications for prediction models, leading to performance degradation and reduced clinical efficacy. Thus, monitoring model shifts and evaluating their impact on prediction models are of utmost importance. Objective: This study aimed to assess the impact of a model shift on ML-based prediction models by evaluating 3 different use cases—delirium, sepsis, and acute kidney injury (AKI)—from 2 hospitals (M and H) with different patient populations and investigate potential model deterioration during the COVID-19 pandemic period. Methods: We trained prediction models using retrospective data from earlier years and examined the presence of a model shift using data from more recent years. We used the area under the receiver operating characteristic curve (AUROC) to evaluate model performance and analyzed the calibration curves over time. We also assessed the influence on clinical decisions by evaluating the alert rate, the rates of over- and underdiagnosis, and the decision curve. Results: The 2 data sets used in this study contained 189,775 and 180,976 medical cases for hospitals M and H, respectively. Statistical analyses (Z test) revealed no significant difference (P>.05) between the AUROCs from the different years for all use cases and hospitals. For example, in hospital M, AKI did not show a significant difference between 2020 (AUROC=0.898) and 2021 (AUROC=0.907, Z=–1.171, P=.242). Similar results were observed in both hospitals and for all use cases (sepsis and delirium) when comparing all the different years. However, when evaluating the calibration curves at the 2 hospitals, model shifts were observed for the delirium and sepsis use cases but not for AKI. Additionally, to investigate the clinical utility of our models, we performed decision curve analysis (DCA) and compared the results across the different years. A pairwise nonparametric statistical comparison showed no differences in the net benefit at the probability thresholds of interest (P>.05). The comprehensive evaluations performed in this study ensured robust model performance of all the investigated models across the years. Moreover, neither performance deteriorations nor alert surges were observed during the COVID-19 pandemic period. Conclusions: Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.


  • Open Access

    Longitudinal Model Shifts of Machine Learning–Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals (Preprint)

    by Patricia Cabanillas Silva, Hong Sun, Mohamed Rezk, Diana M Roccaro-Waldmeyer, Janis Fliegenschmidt, Nikolai Hulde, Vera von Dossow, Laurent Meesseman, Kristof Depraetere, Joerg Stieg, et al.

    BACKGROUND In recent years, machine learning (ML)–based models have been widely used in clinical domains to predict clinical risk events. However, in production, the performances of such models heavily rely on changes in the system and data. The dynamic nature of the system environment, characterized by continuous changes, has significant implications for prediction models, leading to performance degradation and reduced clinical efficacy. Thus, monitoring model shifts and evaluating their impact on prediction models are of utmost importance. OBJECTIVE This study aimed to assess the impact of a model shift on ML-based prediction models by evaluating 3 different use cases—delirium, sepsis, and acute kidney injury (AKI)—from 2 hospitals (M and H) with different patient populations and investigate potential model deterioration during the COVID-19 pandemic period. METHODS We trained prediction models using retrospective data from earlier years and examined the presence of a model shift using data from more recent years. We used the area under the receiver operating characteristic curve (AUROC) to evaluate model performance and analyzed the calibration curves over time. We also assessed the influence on clinical decisions by evaluating the alert rate, the rates of over- and underdiagnosis, and the decision curve. RESULTS The 2 data sets used in this study contained 189,775 and 180,976 medical cases for hospitals M and H, respectively. Statistical analyses (Z test) revealed no significant difference (P>.05) between the AUROCs from the different years for all use cases and hospitals. For example, in hospital M, AKI did not show a significant difference between 2020 (AUROC=0.898) and 2021 (AUROC=0.907, Z=–1.171, P=.242). Similar results were observed in both hospitals and for all use cases (sepsis and delirium) when comparing all the different years. However, when evaluating the calibration curves at the 2 hospitals, model shifts were observed for the delirium and sepsis use cases but not for AKI. Additionally, to investigate the clinical utility of our models, we performed decision curve analysis (DCA) and compared the results across the different years. A pairwise nonparametric statistical comparison showed no differences in the net benefit at the probability thresholds of interest (P>.05). The comprehensive evaluations performed in this study ensured robust model performance of all the investigated models across the years. Moreover, neither performance deteriorations nor alert surges were observed during the COVID-19 pandemic period. CONCLUSIONS Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.


  • Open Access

    Machine Learning–Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live Performance

    by Hong Sun, Kristof Depraetere, Laurent Meesseman, Patricia Cabanillas Silva, Ralph Szymanowsky, Janis Fliegenschmidt, Nikolai Hulde, Vera von Dossow, Martijn Vanbiervliet, Jos De Baerdemaeker, et al.

    Background: Machine learning algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. In this study, we provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. Objective: The main objective of this study was to evaluate clinical risk prediction models in live clinical workflows and compare their performance in these setting with their performance when using retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. Methods: We trained clinical risk prediction models for three use cases (ie, delirium, sepsis, and acute kidney injury) in three different hospitals with retrospective data. We used machine learning and, specifically, deep learning to train models that were based on the Transformer model. The models were trained using a calibration tool that is common for all hospitals and use cases. The models had a common design but were calibrated using each hospital’s specific data. The models were deployed in these three hospitals and used in daily clinical practice. The predictions made by these models were logged and correlated with the diagnosis at discharge. We compared their performance with evaluations on retrospective data and conducted cross-hospital evaluations. Results: The performance of the prediction models with data from live clinical workflows was similar to the performance with retrospective data. The average value of the area under the receiver operating characteristic curve (AUROC) decreased slightly by 0.6 percentage points (from 94.8% to 94.2% at discharge). The cross-hospital evaluations exhibited severely reduced performance: the average AUROC decreased by 8 percentage points (from 94.2% to 86.3% at discharge), which indicates the importance of model calibration with data from the deployment hospital. Conclusions: Calibrating the prediction model with data from different deployment hospitals led to good performance in live settings. The performance degradation in the cross-hospital evaluation identified limitations in developing a generic model for different hospitals. Designing a generic process for model development to generate specialized prediction models for each hospital guarantees model performance in different hospitals.


  • Open Access

    Machine Learning–Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live Performance (Preprint)

    by Hong Sun, Kristof Depraetere, Laurent Meesseman, Patricia Cabanillas Silva, Ralph Szymanowsky, Janis Fliegenschmidt, Nikolai Hulde, Vera von Dossow, Martijn Vanbiervliet, Jos De Baerdemaeker, et al.

    BACKGROUND Machine learning algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. In this study, we provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. OBJECTIVE The main objective of this study was to evaluate clinical risk prediction models in live clinical workflows and compare their performance in these setting with their performance when using retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. METHODS We trained clinical risk prediction models for three use cases (ie, delirium, sepsis, and acute kidney injury) in three different hospitals with retrospective data. We used machine learning and, specifically, deep learning to train models that were based on the Transformer model. The models were trained using a calibration tool that is common for all hospitals and use cases. The models had a common design but were calibrated using each hospital’s specific data. The models were deployed in these three hospitals and used in daily clinical practice. The predictions made by these models were logged and correlated with the diagnosis at discharge. We compared their performance with evaluations on retrospective data and conducted cross-hospital evaluations. RESULTS The performance of the prediction models with data from live clinical workflows was similar to the performance with retrospective data. The average value of the area under the receiver operating characteristic curve (AUROC) decreased slightly by 0.6 percentage points (from 94.8% to 94.2% at discharge). The cross-hospital evaluations exhibited severely reduced performance: the average AUROC decreased by 8 percentage points (from 94.2% to 86.3% at discharge), which indicates the importance of model calibration with data from the deployment hospital. CONCLUSIONS Calibrating the prediction model with data from different deployment hospitals led to good performance in live settings. The performance degradation in the cross-hospital evaluation identified limitations in developing a generic model for different hospitals. Designing a generic process for model development to generate specialized prediction models for each hospital guarantees model performance in different hospitals.

  • Open Access

    Identifying Predictors of Patient Willingness to Have Another Telehealth Consultation in German-Speaking Countries: Standardized Online Questionnaire (Preprint)

    by Diana Manuela Roccaro-Waldmeyer, Philippe Schucht, Winfried Post, Michael Dahlweid

    BACKGROUND Telehealth services can contribute to fulfilling the long-standing promise of patient-centered care by delivering high-quality care at a location convenient to patients or by increasing patient access to healthcare-related information. In German-speaking countries, telehealth consultations were not very common before the outbreak of the Covid-19 pandemic. OBJECTIVE We aimed to investigate the experience of patients in German-speaking countries with telehealth consultations (any medical specialty), especially concerning the willingness of already experienced patients to have another telehealth consultation (primary study outcome) and aspects related to patient empowerment. We hypothesized video consultations to be preferred over phone consultations. METHODS A standardized online questionnaire composed of seven quickly answerable questions did serve to conduct an online survey in line with published recommendations. To estimate the proportion of patients willing to have another telehealth consultation, we determined the necessary minimum sample size to be studied to be 384 (confidence: 95%; error: 5%; population proportion: 50%). RESULTS Our final sample size consists of 411 complete survey responses. For all six aspects captured, respondents having had video consultations rated their telehealth experience much more positively than respondents having had phone consultations. Remarkably, 93% of the respondents having had video consultations (n=204), but only 32% of those having had phone consultations (n=161) (thus, 66% of all 411 respondents having had any telehealth consultation) indicated that they would want to have another telehealth consultation again. By far the most important predictor of this primary outcome was whether patients understood the conversation with their healthcare provider. Notably, 52% of all respondents, respectively 83% of the respondents who have had video consultations, felt more able to make a self-determined decision when comparing their telehealth consultation to a traditional consultation on site. Furthermore, an increased understanding of the conversation with the physician was strongly associated with a higher ability to take a self-determined decision. CONCLUSIONS Our results demonstrate a generally high willingness to repeat telehealth consultations by patients in German-speaking countries and emphasize the importance of patients thoroughly comprehending the communication with their healthcare provider, a prerequisite for patient empowerment. In line with our hypothesis, they further provide evidence for video consultations to lead to higher patient willingness to repeat telehealth consultation, and to a higher degree of patient empowerment, than phone consultations. CLINICALTRIAL

  • Open Access

    Exploring Novel Funding Strategies for Innovative Medical Research: The HORAO Crowdfunding Campaign

    by Philippe Schucht, Diana M Roccaro-Waldmeyer, Michael Murek, Irena Zubak, Johannes Goldberg, Stephanie Falk, Fried-Michael Dahlweid, Andreas Raabe

    Background: The rise of the internet and social media has boosted online crowdfunding as a novel strategy to raise funds for kick-starting projects, but it is rarely used in science. Objective: We report on an online crowdfunding campaign launched in the context of the neuroscience project HORAO. The aim of HORAO was to develop a noninvasive real-time method to visualize neuronal fiber tracts during brain surgery in order to better delineate tumors and to identify crucial cerebral landmarks. The revenue from the crowdfunding campaign was to be used to sponsor a crowdsourcing campaign for the HORAO project. Methods: We ran a 7-week reward-based crowdfunding campaign on a national crowdfunding platform, offering optional material and experiential rewards in return for a contribution toward raising our target of Swiss francs (CHF) 50,000 in financial support (roughly equivalent to US $50,000 at the time of the campaign). We used various owned media (websites and social media), as well as earned media (press releases and news articles) to raise awareness about our project. Results: The production of an explanatory video took 60 hours, and 31 posts were published on social media (Facebook, Instagram, and Twitter). The campaign raised a total of CHF 69,109. Approximately half of all donations came from donors who forwent a reward (CHF 28,786, 48.74%); the other half came from donors who chose experiential and material rewards in similar proportions (CHF 14,958, 25.33% and CHF 15,315.69, 25.93%, respectively). Of those with an identifiable relationship to the crowdfunding team, patients and their relatives contributed the largest sum (CHF 17,820, 30.17%), followed by friends and family (CHF 9288, 15.73%) and work colleagues (CHF 6028, 10.21%), while 43.89% of funds came from donors who were either anonymous or had an unknown relationship to the crowdfunding team. Patients and their relatives made the largest donations, with a median value of CHF 200 (IQR 90). Conclusions: Crowdfunding proved to be a successful strategy to fund a neuroscience project and to raise awareness of a specific clinical problem. Focusing on potential donors with a personal interest in the issue, such as patients and their relatives in our project, is likely to increase funding success. Compared with traditional grant applications, new skills are needed to explain medical challenges to the crowd through video messages and social media.


  • Open Access

    Exploring Novel Funding Strategies for Innovative Medical Research: The HORAO Crowdfunding Campaign (Preprint)

    by Philippe Schucht, Diana M Roccaro-Waldmeyer, Michael Murek, Irena Zubak, Johannes Goldberg, Stephanie Falk, Fried-Michael Dahlweid, Andreas Raabe

    BACKGROUND The rise of the internet and social media has boosted online crowdfunding as a novel strategy to raise funds for kick-starting projects, but it is rarely used in science. OBJECTIVE We report on an online crowdfunding campaign launched in the context of the neuroscience project HORAO. The aim of HORAO was to develop a noninvasive real-time method to visualize neuronal fiber tracts during brain surgery in order to better delineate tumors and to identify crucial cerebral landmarks. The revenue from the crowdfunding campaign was to be used to sponsor a crowdsourcing campaign for the HORAO project. METHODS We ran a 7-week reward-based crowdfunding campaign on a national crowdfunding platform, offering optional material and experiential rewards in return for a contribution toward raising our target of Swiss francs (CHF) 50,000 in financial support (roughly equivalent to US $50,000 at the time of the campaign). We used various owned media (websites and social media), as well as earned media (press releases and news articles) to raise awareness about our project. RESULTS The production of an explanatory video took 60 hours, and 31 posts were published on social media (Facebook, Instagram, and Twitter). The campaign raised a total of CHF 69,109. Approximately half of all donations came from donors who forwent a reward (CHF 28,786, 48.74%); the other half came from donors who chose experiential and material rewards in similar proportions (CHF 14,958, 25.33% and CHF 15,315.69, 25.93%, respectively). Of those with an identifiable relationship to the crowdfunding team, patients and their relatives contributed the largest sum (CHF 17,820, 30.17%), followed by friends and family (CHF 9288, 15.73%) and work colleagues (CHF 6028, 10.21%), while 43.89% of funds came from donors who were either anonymous or had an unknown relationship to the crowdfunding team. Patients and their relatives made the largest donations, with a median value of CHF 200 (IQR 90). CONCLUSIONS Crowdfunding proved to be a successful strategy to fund a neuroscience project and to raise awareness of a specific clinical problem. Focusing on potential donors with a personal interest in the issue, such as patients and their relatives in our project, is likely to increase funding success. Compared with traditional grant applications, new skills are needed to explain medical challenges to the crowd through video messages and social media.

  • Open Access

    Neurons in the Nucleus papilio contribute to the control of eye movements during REM sleep

    by C. Gutierrez Herrera, F. Girard, A. Bilella, T. C. Gent, D. M. Roccaro-Waldmeyer, A. Adamantidis, M. R. Celio

    Rapid eye movements (REM) are characteristic of the eponymous phase of sleep, yet the underlying motor commands remain an enigma. Here, we identified a cluster of Calbindin-D28K-expressing neurons in the Nucleus papilio (NPCalb), located in the dorsal paragigantocellular nucleus, which are active during REM sleep and project to the three contralateral eye-muscle nuclei. The firing of opto-tagged NPCalb neurons is augmented prior to the onset of eye movements during REM sleep. Optogenetic activation of NPCalb neurons triggers eye movements selectively during REM sleep, while their genetic ablation or optogenetic silencing suppresses them. None of these perturbations led to a change in the duration of REM sleep episodes. Our study provides the first evidence for a brainstem premotor command contributing to the control of eye movements selectively during REM sleep in the mammalian brain.