Stefanie Painter

Background: The growth in the capabilities of telehealth have made it possible to identify individuals with a higher risk of uncontrolled diabetes and provide them with targeted support and resources to help them manage their condition. Thus, predictive modeling has emerged as a valuable tool for the advancement of diabetes management. Objective: This study aimed to conceptualize and develop a novel machine learning (ML) approach to proactively identify participants enrolled in a remote diabetes monitoring program (RDMP) who were at risk of uncontrolled diabetes at 12 months in the program. Methods: Registry data from the Livongo for Diabetes RDMP were used to design separate dynamic predictive ML models to predict participant outcomes at each monthly checkpoint of the participants’ program journey (month-n models) from the first day of onboarding (month-0 model) up to the 11th month (month-11 model). A participant’s program journey began upon onboarding into the RDMP and monitoring their own blood glucose (BG) levels through the RDMP-provided BG meter. Each participant passed through 12 predicative models through their first year enrolled in the RDMP. Four categories of participant attributes (ie, survey data, BG data, medication fills, and health signals) were used for feature construction. The models were trained using the light gradient boosting machine and underwent hyperparameter tuning. The performance of the models was evaluated using standard metrics, including precision, recall, specificity, the area under the curve, the F1-score, and accuracy. Results: The ML models exhibited strong performance, accurately identifying observable at-risk participants, with recall ranging from 70% to 94% and precision from 40% to 88% across the 12-month program journey. Unobservable at-risk participants also showed promising performance, with recall ranging from 61% to 82% and precision from 42% to 61%. Overall, model performance improved as participants progressed through their program journey, demonstrating the importance of engagement data in predicting long-term clinical outcomes. Conclusions: This study explored the Livongo for Diabetes RDMP participants’ temporal and static attributes, identification of diabetes management patterns and characteristics, and their relationship to predict diabetes management outcomes. Proactive targeting ML models accurately identified participants at risk of uncontrolled diabetes with a high level of precision that was generalizable through future years within the RDMP. The ability to identify participants who are at risk at various time points throughout the program journey allows for personalized interventions to improve outcomes. This approach offers significant advancements in the feasibility of large-scale implementation in remote monitoring programs and can help prevent uncontrolled glycemic levels and diabetes-related complications. Future research should include the impact of significant changes that can affect a participant’s diabetes management.

BACKGROUND The growth in the capabilities of telehealth have made it possible to identify individuals with a higher risk of uncontrolled diabetes and provide them with targeted support and resources to help them manage their condition. Thus, predictive modeling has emerged as a valuable tool for the advancement of diabetes management. OBJECTIVE This study aimed to conceptualize and develop a novel machine learning (ML) approach to proactively identify participants enrolled in a remote diabetes monitoring program (RDMP) who were at risk of uncontrolled diabetes at 12 months in the program. METHODS Registry data from the Livongo for Diabetes RDMP were used to design separate dynamic predictive ML models to predict participant outcomes at each monthly checkpoint of the participants’ program journey (month-n models) from the first day of onboarding (month-0 model) up to the 11th month (month-11 model). A participant’s program journey began upon onboarding into the RDMP and monitoring their own blood glucose (BG) levels through the RDMP-provided BG meter. Each participant passed through 12 predicative models through their first year enrolled in the RDMP. Four categories of participant attributes (ie, survey data, BG data, medication fills, and health signals) were used for feature construction. The models were trained using the light gradient boosting machine and underwent hyperparameter tuning. The performance of the models was evaluated using standard metrics, including precision, recall, specificity, the area under the curve, the F1-score, and accuracy. RESULTS The ML models exhibited strong performance, accurately identifying observable at-risk participants, with recall ranging from 70% to 94% and precision from 40% to 88% across the 12-month program journey. Unobservable at-risk participants also showed promising performance, with recall ranging from 61% to 82% and precision from 42% to 61%. Overall, model performance improved as participants progressed through their program journey, demonstrating the importance of engagement data in predicting long-term clinical outcomes. CONCLUSIONS This study explored the Livongo for Diabetes RDMP participants’ temporal and static attributes, identification of diabetes management patterns and characteristics, and their relationship to predict diabetes management outcomes. Proactive targeting ML models accurately identified participants at risk of uncontrolled diabetes with a high level of precision that was generalizable through future years within the RDMP. The ability to identify participants who are at risk at various time points throughout the program journey allows for personalized interventions to improve outcomes. This approach offers significant advancements in the feasibility of large-scale implementation in remote monitoring programs and can help prevent uncontrolled glycemic levels and diabetes-related complications. Future research should include the impact of significant changes that can affect a participant’s diabetes management.

Background: Evidence-based digital health programs have shown efficacy in being primary tools to improve emotional and mental health, as well as offering supplementary support to individuals undergoing psychotherapy for anxiety, depression, and other mental health disorders. However, information is lacking about the dose response to digital mental health interventions. Objective: The objective of the study was to examine the effect of time in program and program usage on symptom change among individuals enrolled in a real-world comprehensive digital mental health program (myStrength) who are experiencing severe anxiety or depression. Methods: Eligible participants (N=18,626) were adults aged 18 years and older who were enrolled in myStrength for at least four weeks as part of their employee wellness benefit program, who completed baseline, the 2-week, 2-month, and 6-month surveys querying symptoms of anxiety (Generalized Anxiety Disorder–7 [GAD-7]) and depression (Patient Health Questionnaire–9 [PHQ-9]). Linear growth curve models were used to analyze the effect of average weekly program usage on subsequent GAD-7 and PHQ-9 scores for participants with scores indicating severe anxiety (GAD-7≥15) or depression (PHQ-9≥15). All models were adjusted for baseline score and demographics. Results: Participants in the study (N=1519) were 77.4% female (1176/1519), had a mean age of 45 years (SD 14 years), and had an average enrollment time of 3 months. At baseline, participants reported an average of 9.39 (SD 6.04) on the GAD-7 and 11.0 (SD 6.6) on the PHQ-9. Those who reported 6-month results had an average of 8.18 (SD 6.15) on the GAD-7 and 9.18 (SD 6.79) on the PHQ-9. Participants with severe scores (n=506) experienced a significant improvement of 2.97 (SE 0.35) and 3.97 (SE 0.46) at each time point for anxiety and depression, respectively (t=–8.53 and t=–8.69, respectively; Ps<.001). Those with severe baseline scores also saw a reduction of 0.27 (SE 0.08) and 0.25 (SE 0.09) points in anxiety and depression, respectively, for each additional program activity per week (t=–3.47 and t=–2.66, respectively; Ps<.05). Conclusions: For participants with severe baseline scores, the study found a clinically significant reduction of approximately 9 points for anxiety and 12 points for depression after 6 months of enrollment, suggesting that interventions targeting mental health must maintain active, ongoing engagement when symptoms are present and be available as a continuous resource to maximize clinical impact, specifically in those experiencing severe anxiety or depression. Moreover, a dosing effect was shown, indicating improvement in outcomes among participants who engaged with the program every other day for both anxiety and depression. This suggests that digital mental health programs that provide both interesting and evidence-based activities could be more successful in further improving mental health outcomes.

Background: Diabetes management is complex, and program personalization has been identified to enhance engagement and clinical outcomes in diabetes management programs. However, 50% of individuals living with diabetes are unable to achieve glycemic control, presenting a gap in the delivery of self-management education and behavior change. Machine learning and recommender systems, which have been used within the health care setting, could be a feasible application for diabetes management programs to provide a personalized user experience and improve user engagement and outcomes. Objective: This study aims to evaluate machine learning models using member-level engagements to predict improvement in estimated A1c and develop personalized action recommendations within a remote diabetes monitoring program to improve clinical outcomes. Methods: A retrospective study of Livongo for Diabetes member engagement data was analyzed within five action categories (interacting with a coach, reading education content, self-monitoring blood glucose level, tracking physical activity, and monitoring nutrition) to build a member-level model to predict if a specific type and level of engagement could lead to improved estimated A1c for members with type 2 diabetes. Engagement and improvement in estimated A1c can be correlated; therefore, the doubly robust learning method was used to model the heterogeneous treatment effect of action engagement on improvements in estimated A1c. Results: The treatment effect was successfully computed within the five action categories on estimated A1c reduction for each member. Results show interaction with coaches and self-monitoring blood glucose levels were the actions that resulted in the highest average decrease in estimated A1c (1.7% and 1.4%, respectively) and were the most recommended actions for 54% of the population. However, these were found to not be the optimal interventions for all members; 46% of members were predicted to have better outcomes with one of the other three interventions. Members who engaged with their recommended actions had on average a 0.8% larger reduction in estimated A1c than those who did not engage in recommended actions within the first 3 months of the program. Conclusions: Personalized action recommendations using heterogeneous treatment effects to compute the impact of member actions can reduce estimated A1c and be a valuable tool for diabetes management programs in encouraging members toward actions to improve clinical outcomes.

BACKGROUND Evidence-based digital health programs have shown efficacy in being primary tools to improve emotional and mental health, as well as offering supplementary support to individuals undergoing psychotherapy for anxiety, depression, and other mental health disorders. However, information is lacking about the dose response to digital mental health interventions. OBJECTIVE The objective of the study was to examine the effect of time in program and program usage on symptom change among individuals enrolled in a real-world comprehensive digital mental health program (myStrength) who are experiencing severe anxiety or depression. METHODS Eligible participants (N=18,626) were adults aged 18 years and older who were enrolled in myStrength for at least four weeks as part of their employee wellness benefit program, who completed baseline, the 2-week, 2-month, and 6-month surveys querying symptoms of anxiety (Generalized Anxiety Disorder–7 [GAD-7]) and depression (Patient Health Questionnaire–9 [PHQ-9]). Linear growth curve models were used to analyze the effect of average weekly program usage on subsequent GAD-7 and PHQ-9 scores for participants with scores indicating severe anxiety (GAD-7≥15) or depression (PHQ-9≥15). All models were adjusted for baseline score and demographics. RESULTS Participants in the study (N=1519) were 77.4% female (1176/1519), had a mean age of 45 years (SD 14 years), and had an average enrollment time of 3 months. At baseline, participants reported an average of 9.39 (SD 6.04) on the GAD-7 and 11.0 (SD 6.6) on the PHQ-9. Those who reported 6-month results had an average of 8.18 (SD 6.15) on the GAD-7 and 9.18 (SD 6.79) on the PHQ-9. Participants with severe scores (n=506) experienced a significant improvement of 2.97 (SE 0.35) and 3.97 (SE 0.46) at each time point for anxiety and depression, respectively (t=–8.53 and t=–8.69, respectively; Ps<.001). Those with severe baseline scores also saw a reduction of 0.27 (SE 0.08) and 0.25 (SE 0.09) points in anxiety and depression, respectively, for each additional program activity per week (t=–3.47 and t=–2.66, respectively; Ps<.05). CONCLUSIONS For participants with severe baseline scores, the study found a clinically significant reduction of approximately 9 points for anxiety and 12 points for depression after 6 months of enrollment, suggesting that interventions targeting mental health must maintain active, ongoing engagement when symptoms are present and be available as a continuous resource to maximize clinical impact, specifically in those experiencing severe anxiety or depression. Moreover, a dosing effect was shown, indicating improvement in outcomes among participants who engaged with the program every other day for both anxiety and depression. This suggests that digital mental health programs that provide both interesting and evidence-based activities could be more successful in further improving mental health outcomes.

Regular physical activity (PA) has been shown to improve glycemic control in persons with type 2 diabetes. This study aimed to investigate the impact of PA on blood glucose after controlling for medication use, demographics, and week of activation using a real-world population of individuals with type 2 diabetes. A longitudinal, retrospective study was performed evaluating weekly PA of Livongo members (N = 9,509), which analyzed fasting blood glucose (FBG), step counts, and daily active minutes. Linear mixed-effect modeling technique was used to investigate within member and between member effects of input variables on average weekly FBG. Of members enrolled, 6,336 (32%) had self-reported body mass index, qualified week with diabetes medications, and FBG measures. Members’ baseline average age was 49.4 (SD 10.1) years old, 43% female, and 45,496 member weeks with an average of 7.2 qualified weeks (PA observable in ≥4 days) per member. Average weekly FBG was 140.5 mg/dL (SD 39.8), and average daily step counts were 4,833 (SD 3,266). Moving from sedentary (<5,000 steps per day) to active (≥5,000 steps per day) resulted in mean weekly FBG reduction of 13 mg/dL (95% CI: −22.6 to −3.14). One additional day of ≥8,000 steps reduced mean weekly FBG by 0.47 mg/dL (95% CI: −0.77 to −0.16). Members who completed 30 min of moderate to vigorous PA above the population average reduced mean weekly FBG by 7.7 mg/dL (95% CI: −13.4 to −2.0). PA is associated with a mean weekly FBG reduction of 13 mg/dL when changing from a sedentary to active lifestyle while participating in a remote diabetes monitoring program.

BACKGROUND Diabetes management is complex, and program personalization has been identified to enhance engagement and clinical outcomes in diabetes management programs. However, 50% of individuals living with diabetes are unable to achieve glycemic control, presenting a gap in the delivery of self-management education and behavior change. Machine learning and recommender systems, which have been used within the health care setting, could be a feasible application for diabetes management programs to provide a personalized user experience and improve user engagement and outcomes. OBJECTIVE This study aims to evaluate machine learning models using member-level engagements to predict improvement in estimated A1c and develop personalized action recommendations within a remote diabetes monitoring program to improve clinical outcomes. METHODS A retrospective study of Livongo for Diabetes member engagement data was analyzed within five action categories (interacting with a coach, reading education content, self-monitoring blood glucose level, tracking physical activity, and monitoring nutrition) to build a member-level model to predict if a specific type and level of engagement could lead to improved estimated A1c for members with type 2 diabetes. Engagement and improvement in estimated A1c can be correlated; therefore, the doubly robust learning method was used to model the heterogeneous treatment effect of action engagement on improvements in estimated A1c. RESULTS The treatment effect was successfully computed within the five action categories on estimated A1c reduction for each member. Results show interaction with coaches and self-monitoring blood glucose levels were the actions that resulted in the highest average decrease in estimated A1c (1.7% and 1.4%, respectively) and were the most recommended actions for 54% of the population. However, these were found to not be the optimal interventions for all members; 46% of members were predicted to have better outcomes with one of the other three interventions. Members who engaged with their recommended actions had on average a 0.8% larger reduction in estimated A1c than those who did not engage in recommended actions within the first 3 months of the program. CONCLUSIONS Personalized action recommendations using heterogeneous treatment effects to compute the impact of member actions can reduce estimated A1c and be a valuable tool for diabetes management programs in encouraging members toward actions to improve clinical outcomes.

Emerging Technologies, Medical Devices, Apps, Sensors, and Informatics to Help People with Diabetes

Introduction To investigate the impact of the digital Livongo Diabetes Prevention Program (DPP) on weight at 12 months, understand participants’ self-monitoring behaviors associated with greater weight loss, and evaluate the impact of coaching interactions on more frequent self-monitoring behaviors. Research design and methods A retrospective analysis was performed using data from 2037 participants enrolled in the Livongo DPP who completed lesson 1 and recorded a starting weight during 2016–2017. Self-monitoring behaviors, including weigh-ins, food logging, activity, and coach–participant interactions, were analyzed at 6 and 12 months. Subgroup analysis was conducted based on those who were highly engaged versus those minimally engaged. Multiple regression analysis was performed using demographic, self-monitoring, and lesson attendance data to determine predictors of weight loss at 12 months and coaching impact on self-monitoring. Results Participants had a mean age of 50 years (SD ±12), with a starting weight of 94 kg (SD ±21), were college-educated (78%), and were female (74%). Overall, participants lost on average 5.1% of their starting weight. Highly engaged participants lost 6.6% of starting body weight, with 25% losing ≥10% at 12 months. Logistic regression analysis showed each submitted food log was associated with 0.23 kg (pConclusions Food logging had the largest impact on weight loss, followed by lesson engagement and physical activity. Future studies should examine further opportunities to deliver nutrition-based content to increase and sustain weight loss for DPP.

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).