ILIlya Lipkovich

Ilya Lipkovich

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Ilya Lipkovich has a total of 74 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Next-Generation Sequencing–Based Testing Among Patients With Advanced or Metastatic Nonsquamous Non–Small Cell Lung Cancer in the United States: Predictive Modeling Using Machine Learning Methods

    by Alan James Michael Brnabic, Ilya Lipkovich, Zbigniew Kadziola, Dan He, Peter M Krein, Lisa M Hess

    Background: Next-generation sequencing (NGS) has become a cornerstone of treatment for lung cancer and is recommended in current treatment guidelines for patients with advanced or metastatic disease. Objective: This study was designed to use machine learning methods to determine demographic and clinical characteristics of patients with advanced or metastatic non–small cell lung cancer (NSCLC) that may predict likelihood of receiving NGS-based testing (ever vs never NGS-tested) as well as likelihood of timing of testing (early vs late NGS-tested). Methods: Deidentified patient-level data were analyzed in this study from a real-world cohort of patients with advanced or metastatic NSCLC in the United States. Patients with nonsquamous disease, who received systemic therapy for NSCLC, and had at least 3 months of follow-up data for analysis were included in this study. Three strategies, logistic regression models, penalized logistic regression using least absolute shrinkage and selection operator penalty, and extreme gradient boosting with classification trees as base learners, were used to identify predictors of ever versus never and early versus late NGS testing. Data were split into D1 (training+validation; 80%) and D2 (testing; 20%) sets; the 3 strategies were evaluated by comparing their performance on multiple m=1000 splits in the training (70%) and validation data (30%) within the D1 set. The final model was selected by evaluating performance using the area under the receiver operating curve while taking into account considerations of simplicity and clinical interpretability. Performance was re-estimated using the test data D2. Results: A total of 13,425 met the criteria for the ever NGS-tested, and 17,982 were included in the never NGS-tested group. Performance metrics showed the area under the receiver operating curve evaluated from validation data was similar across all models (77%-84%). Among those in the ever NGS-tested group, 84.08% (n=11,289) were early NGS-tested, and 15.91% (n=2136) late NGS-tested. Factors associated with both ever having NGS testing as well as early NGS testing included later year of NSCLC diagnosis, no smoking history, and evidence of programmed death ligand 1 testing (all P<.05). Factors associated with a greater chance of never receiving NGS testing included older age, lower performance status, Black race, higher number of single-gene tests, public insurance, and treatment in a geography with Molecular Diagnostics Services Program adoption (all P<.05). Conclusions: Predictors of ever versus never as well as early versus late NGS testing in the setting of advanced or metastatic NSCLC were consistent across machine learning methods in this study, demonstrating the ability of these models to identify factors that may predict NGS-based testing. There is a need to ensure that patients regardless of age, race, insurance status, and geography (factors associated with lower odds of receiving NGS testing in this study) are provided with equitable access to NGS-based testing.


  • Open Access

    Analytic Challenges in Clinical Trials of Early Alzheimer’s Disease

    by Craig H. Mallinckrodt, Ilya Lipkovich, Suzanne B. Hendrix, Samuel P. Dickson, Geert Molenberghs

    Background: The heavily right-skewed data seen in recently reported Alzheimer’s disease (AD) clinical trials influenced treatment contrasts when data were analyzed via the typical mixed-effects model for repeated measures (MMRM). Methods: An MMRM analysis similar to what is commonly used in AD clinical trials was compared versus robust regression (RR) and the non-parametric Hodges–Lehman estimator (HL). Results: Results in simulated data patterned after AD trials showed that imbalance across treatment arms in the number of patients in the extreme right tail (those with rapid disease progression) frequently occurred. Each analysis method controlled Type I error at or below the nominal level. The RR analysis yielded smaller standard errors and more power than MMRM and HL. In data sets with appreciable imbalance in the number of rapid progressing patients, MMRM results favored the treatment arm with fewer rapid progressors. Results from HL showed the same trend but to a lesser degree. Robust regression yielded similar results regardless of the ratio of rapid progressors. Conclusions: Although more research is needed over a wider range of scenarios, it should not be assumed that MMRM is the optimal approach for trials in early Alzheimer’s disease.

  • Modern approaches for evaluating treatment effect heterogeneity from clinical trials and observational data

    by Ilya Lipkovich, David Svensson, Bohdana Ratitch, Alex Dmitrienko

    In this paper, we review recent advances in statistical methods for the evaluation of the heterogeneity of treatment effects (HTE), including subgroup identification and estimation of individualized treatment regimens, from randomized clinical trials and observational studies. We identify several types of approaches using the features introduced in Lipkovich et al (Stat Med 2017;36: 136‐196) that distinguish the recommended principled methods from basic methods for HTE evaluation that typically rely on rules of thumb and general guidelines (the methods are often referred to as common practices). We discuss the advantages and disadvantages of various principled methods as well as common measures for evaluating their performance. We use simulated data and a case study based on a historical clinical trial to illustrate several new approaches to HTE evaluation.


  • Open Access

    Next-Generation Sequencing–Based Testing Among Patients With Advanced or Metastatic Nonsquamous Non–Small Cell Lung Cancer in the United States: Predictive Modeling Using Machine Learning Methods (Preprint)

    by Alan James Michael Brnabic, Ilya Lipkovich, Zbigniew Kadziola, Dan He, Peter M Krein, Lisa M Hess

    BACKGROUND Next-generation sequencing (NGS) has become a cornerstone of treatment for lung cancer and is recommended in current treatment guidelines for patients with advanced or metastatic disease. OBJECTIVE This study was designed to use machine learning methods to determine demographic and clinical characteristics of patients with advanced or metastatic non–small cell lung cancer (NSCLC) that may predict likelihood of receiving NGS-based testing (ever vs never NGS-tested) as well as likelihood of timing of testing (early vs late NGS-tested). METHODS Deidentified patient-level data were analyzed in this study from a real-world cohort of patients with advanced or metastatic NSCLC in the United States. Patients with nonsquamous disease, who received systemic therapy for NSCLC, and had at least 3 months of follow-up data for analysis were included in this study. Three strategies, logistic regression models, penalized logistic regression using least absolute shrinkage and selection operator penalty, and extreme gradient boosting with classification trees as base learners, were used to identify predictors of ever versus never and early versus late NGS testing. Data were split into D1 (training+validation; 80%) and D2 (testing; 20%) sets; the 3 strategies were evaluated by comparing their performance on multiple m=1000 splits in the training (70%) and validation data (30%) within the D1 set. The final model was selected by evaluating performance using the area under the receiver operating curve while taking into account considerations of simplicity and clinical interpretability. Performance was re-estimated using the test data D2. RESULTS A total of 13,425 met the criteria for the ever NGS-tested, and 17,982 were included in the never NGS-tested group. Performance metrics showed the area under the receiver operating curve evaluated from validation data was similar across all models (77%-84%). Among those in the ever NGS-tested group, 84.08% (n=11,289) were early NGS-tested, and 15.91% (n=2136) late NGS-tested. Factors associated with both ever having NGS testing as well as early NGS testing included later year of NSCLC diagnosis, no smoking history, and evidence of programmed death ligand 1 testing (all P<.05). Factors associated with a greater chance of never receiving NGS testing included older age, lower performance status, Black race, higher number of single-gene tests, public insurance, and treatment in a geography with Molecular Diagnostics Services Program adoption (all P<.05). CONCLUSIONS Predictors of ever versus never as well as early versus late NGS testing in the setting of advanced or metastatic NSCLC were consistent across machine learning methods in this study, demonstrating the ability of these models to identify factors that may predict NGS-based testing. There is a need to ensure that patients regardless of age, race, insurance status, and geography (factors associated with lower odds of receiving NGS testing in this study) are provided with equitable access to NGS-based testing.

  • Correction to “Using principal stratification in analysis of clinical trials”

    by Ilya Lipkovich, Bohdana Ratitch, Yongming Qu, Xiang Zhang, Mingyang Shan, Craig Mallinckrodt

  • Open Access

    Participant Engagement and Adherence to Providing Smartwatch and Patient-Reported Outcome Data: Digital Tracking of Rheumatoid Arthritis Longitudinally (DIGITAL) Real-World Study

    by William B Nowell, Jeffrey R Curtis, Hong Zhao, Fenglong Xie, Laura Stradford, David Curtis, Kelly Gavigan, Jessica Boles, Cassie Clinton, Ilya Lipkovich, et al.

    Background: Digital health studies using electronic patient-reported outcomes (ePROs) and wearables bring new challenges, including the need for participants to consistently provide trial data. Objective: This study aims to characterize the engagement, protocol adherence, and data completeness among participants with rheumatoid arthritis enrolled in the Digital Tracking of Arthritis Longitudinally (DIGITAL) study. Methods: Participants were invited to participate in this app-based study, which included a 14-day run-in and an 84-day main study. In the run-in period, data were collected via the ArthritisPower mobile app to increase app familiarity and identify the individuals who were motivated to participate. Successful completers of the run-in period were mailed a wearable smartwatch, and automated and manual prompts were sent to participants, reminding them to complete app input or regularly wear and synchronize devices, respectively, during the main study. Study coordinators monitored participant data and contacted participants via email, SMS text messaging, and phone to resolve adherence issues per a priori rules, in which consecutive spans of missing data triggered participant contact. Adherence to data collection during the main study period was defined as providing requested data for >70% of 84 days (daily ePRO, ≥80% daily smartwatch data) or at least 9 of 12 weeks (weekly ePRO). Results: Of the 470 participants expressing initial interest, 278 (59.1%) completed the run-in period and qualified for the main study. Over the 12-week main study period, 87.4% (243/278) of participants met the definition of adherence to protocol-specified data collection for weekly ePRO, and 57.2% (159/278) did so for daily ePRO. For smartwatch data, 81.7% (227/278) of the participants adhered to the protocol-specified data collection. In total, 52.9% (147/278) of the participants met composite adherence. Conclusions: Compared with other digital health rheumatoid arthritis studies, a short run-in period appears useful for identifying participants likely to engage in a study that collects data via a mobile app and wearables and gives participants time to acclimate to study requirements. Automated or manual prompts (ie, “It’s time to sync your smartwatch”) may be necessary to optimize adherence. Adherence varies by data collection type (eg, ePRO vs smartwatch data). International Registered Report Identifier (IRRID): RR2-10.2196/14665


  • Overview of modern approaches for identifying and evaluating heterogeneous treatment effects from clinical data

    by Ilya Lipkovich, David Svensson, Bohdana Ratitch, Alex Dmitrienko

    There has been much interest in the evaluation of heterogeneous treatment effects (HTE) and multiple statistical methods have emerged under the heading of personalized/precision medicine combining ideas from hypothesis testing, causal inference, and machine learning over the past 10-15 years. We discuss new ideas and approaches for evaluating HTE in randomized clinical trials and observational studies using the features introduced earlier by Lipkovich, Dmitrienko, and D’Agostino that distinguish principled methods from simplistic approaches to data-driven subgroup identification and estimating individual treatment effects and use a case study to illustrate these approaches. We identified and provided a high-level overview of several classes of modern statistical approaches for personalized/precision medicine, elucidated the underlying principles and challenges, and compared findings for a case study across different methods. Different approaches to evaluating HTEs may produce (and actually produced) highly disparate results when applied to a specific data set. Evaluating HTE with machine learning methods presents special challenges since most of machine learning algorithms are optimized for prediction rather than for estimating causal effects. An additional challenge is in that the output of machine learning methods is typically a “black box” that needs to be transformed into interpretable personalized solutions in order to gain acceptance and usability.


  • Open Access

    Participant Engagement and Adherence to Providing Smartwatch and Patient-Reported Outcome Data: Digital Tracking of Rheumatoid Arthritis Longitudinally (DIGITAL) Real-World Study (Preprint)

    by William B Nowell, Jeffrey R Curtis, Hong Zhao, Fenglong Xie, Laura Stradford, David Curtis, Kelly Gavigan, Jessica Boles, Cassie Clinton, Ilya Lipkovich, et al.

    BACKGROUND Digital health studies using electronic patient-reported outcomes (ePROs) and wearables bring new challenges, including the need for participants to consistently provide trial data. OBJECTIVE This study aims to characterize the engagement, protocol adherence, and data completeness among participants with rheumatoid arthritis enrolled in the Digital Tracking of Arthritis Longitudinally (DIGITAL) study. METHODS Participants were invited to participate in this app-based study, which included a 14-day run-in and an 84-day main study. In the run-in period, data were collected via the ArthritisPower mobile app to increase app familiarity and identify the individuals who were motivated to participate. Successful completers of the run-in period were mailed a wearable smartwatch, and automated and manual prompts were sent to participants, reminding them to complete app input or regularly wear and synchronize devices, respectively, during the main study. Study coordinators monitored participant data and contacted participants via email, SMS text messaging, and phone to resolve adherence issues per a priori rules, in which consecutive spans of missing data triggered participant contact. Adherence to data collection during the main study period was defined as providing requested data for >70% of 84 days (daily ePRO, ≥80% daily smartwatch data) or at least 9 of 12 weeks (weekly ePRO). RESULTS Of the 470 participants expressing initial interest, 278 (59.1%) completed the run-in period and qualified for the main study. Over the 12-week main study period, 87.4% (243/278) of participants met the definition of adherence to protocol-specified data collection for weekly ePRO, and 57.2% (159/278) did so for daily ePRO. For smartwatch data, 81.7% (227/278) of the participants adhered to the protocol-specified data collection. In total, 52.9% (147/278) of the participants met composite adherence. CONCLUSIONS Compared with other digital health rheumatoid arthritis studies, a short run-in period appears useful for identifying participants likely to engage in a study that collects data via a mobile app and wearables and gives participants time to acclimate to study requirements. Automated or manual prompts (ie, “It’s time to sync your smartwatch”) may be necessary to optimize adherence. Adherence varies by data collection type (eg, ePRO vs smartwatch data). INTERNATIONAL REGISTERED REPORT RR2-10.2196/14665

  • Open Access

    Using principal stratification in analysis of clinical trials

    by Ilya Lipkovich, Bohdana Ratitch, Yongming Qu, Xiang Zhang, Mingyang Shan, Craig Mallinckrodt

    The ICH E9(R1) addendum (2019) proposed principal stratification (PS) as one of five strategies for dealing with intercurrent events. Therefore, understanding the strengths, limitations, and assumptions of PS is important for the broad community of clinical trialists. Many approaches have been developed under the general framework of PS in different areas of research, including experimental and observational studies. These diverse applications have utilized a diverse set of tools and assumptions. Thus, need exists to present these approaches in a unifying manner. The goal of this tutorial is threefold. First, we provide a coherent and unifying description of PS. Second, we emphasize that estimation of effects within PS relies on strong assumptions and we thoroughly examine the consequences of these assumptions to understand in which situations certain assumptions are reasonable. Finally, we provide an overview of a variety of key methods for PS analysis and use a real clinical trial example to illustrate them. Examples of code for implementation of some of these approaches are given in supplemental materials.