Development and Validation of Forecasting Next Reported Seizure Using e‐Diaries
- 9 July 2020
- journal article
- research article
- Published by Wiley in Annals of Neurology
- Vol. 88 (3) , 588-595
Abstract
Objective: There are no validated methods for predicting the timing of seizures. Using machine learning, we sought to forecast 24‐hour risk of self‐reported seizure from e‐diaries.Methods: Data from 5,419 patients on SeizureTracker.com (including seizure count, type, and duration) were split into training (3,806 patients/1,665,215 patient‐days) and testing (1,613 patients/549,588 patient‐days) sets with no overlapping patients. An artificial intelligence (AI) program, consisting of recurrent networks followed by a multilayer perceptron (“deep learning” model), was trained to produce risk forecasts. Forecasts were made from a sliding window of 3‐month diary history for each day of each patient's diary. After training, the model parameters were held constant and the testing set was scored. A rate‐matched random (RMR) forecast was compared to the AI. Comparisons were made using the area under the receiver operating characteristic curve (AUC), a measure of binary discrimination performance, and the Brier score, a measure of forecast calibration. The Brier skill score (BSS) measured the improvement of the AI Brier score compared to the benchmark RMR Brier score. Confidence intervals (CIs) on performance statistics were obtained via bootstrapping.Results: The AUC was 0.86 (95% CI = 0.85–0.88) for AI and 0.83 (95% CI = 0.81–0.85) for RMR, favoring AI (p < 0.001). Overall (all patients combined), BSS was 0.27 (95% CI = 0.23–0.31), also favoring AI (p < 0.001).Interpretation: The AI produced a valid forecast superior to a chance forecaster, and provided meaningful forecasts in the majority of patients. Future studies will be needed to quantify the clinical value of these forecasts for patients. ANN NEUROL 2020;88:588–595Keywords
Funding Information
- Beth Israel Deaconess Medical Center (Department of neurology grant)
- National Institute of Neurological Disorders and Stroke (K23NS090900, R01NS102190, R01NS107291, T32NS048005)
- Harvard Medical School
This publication has 30 references indexed in Scilit:
- Seizure Prediction: An Approach Using Probabilistic Forecasting2016
- Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD StatementEuropean Urology, 2015
- Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and ElaborationAnnals of Internal Medicine, 2015
- Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man studyThe Lancet Neurology, 2013
- Seizure diaries for clinical research and practice: Limitations and future prospectsEpilepsy & Behavior, 2012
- Management of refractory status epilepticus in adults: still more questions than answersThe Lancet Neurology, 2011
- The statistics of a practical seizure warning systemJournal of Neural Engineering, 2008
- Can patients with epilepsy predict their seizures?Neurology, 2007
- Testing statistical significance of multivariate time series analysis techniques for epileptic seizure predictionChaos: An Interdisciplinary Journal of Nonlinear Science, 2006
- VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITYMonthly Weather Review, 1950