Qi Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Qi Liu has a total of 358 co-authors. Here are the top co-authors by number of shared publications:
- Shiew-Mei Huang
Office of Clinical Pharmacology, Office of Translational Sciences Center for Drug Evaluation and Research, US Food and Drug Administration (OCP/CDER/FDA) Silver Spring Maryland USA
11 shared publications - Hao Zhu
Office of Clinical Pharmacology Office of Translational Sciences, Center for Drug Evaluation and Research US Food and Drug Administration Silver Spring MD USA
10 shared publications - Ruihao Huang
Office of Clinical Pharmacology, Center of Drug Evaluation and Research US Food and Drug Administration Silver Spring Maryland USA
9 shared publications Hao Zhu
Office of Translational Sciences, Center for Drug Evaluation and Research U.S. Food and Drug Administration Silver Spring Maryland USA
7 shared publications- Chenan Zhang
Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California
5 shared publications - Donna R. Rivera
Oncology Center of Excellence, US Food and Drug Administration, Silver Spring, MD
5 shared publications
Latest Publications
- Open Access
- 21 October 2026
- Published by Wiley in Clinical and Translational Science
- Vol. 19 (11) , e70750
Landscape Analysis of Digital Health Technology Use for Remote Data Acquisition in CDER ‐Regulated Submissions, 2023–2025
Digital health technologies can support remote acquisition of clinical investigation data and may facilitate more flexible clinical trial designs. In March 2023, FDA revised Forms FDA 1571 and FDA 356h to include a structured yes/no field asking whether a submission contains DHT data or a proposal to collect DHT data. This change created a mechanism for identifying submissions involving DHT‐based remote data acquisition. Following implementation of these fields, the Office of Medical Policy in FDA's Center for Drug Evaluation and Research developed and implemented a systematic process to track, identify, and adjudicate DHT‐related submissions. This analytical report presents a descriptive, aggregated, de‐identified analysis of submissions that were flagged through this operational process from March 2023 through December 2025. Operational inclusion criteria were applied to identify submissions involving DHTs used for remote data acquisition intended to support regulatory decision‐making. Of 2318 flagged submissions, 277 (12%) were confirmed for inclusion in this analysis. Confirmed submissions were predominantly associated with investigational applications (237/277, 86%). When classified by the nature of data collected, submissions reflected a range of remote data acquisition approaches, including patient reported data and sensor‐based measurements, with some submissions combining more than one modality. Electronic diary and electronic patient‐reported outcomes were most common, exclusively appearing in 161 submissions (58%), followed by data captured from general objective sensors in 92 submissions (33%) and continuous glucose monitoring data in 24 submissions (9%). These findings establish a quantitative baseline for DHT‐based remote data acquisition in CDER submissions and the implementation of FDA's corresponding guidance.
- 21 October 2026
- Published by Wiley in Clinical and Translational Science
- Vol. 19 (11) , e70750
Open AccessLandscape Analysis of Digital Health Technology Use for Remote Data Acquisition in CDER ‐Regulated Submissions, 2023–2025
Digital health technologies can support remote acquisition of clinical investigation data and may facilitate more flexible clinical trial designs. In March 2023, FDA revised Forms FDA 1571 and FDA 356h to include a structured yes/no field asking whether a submission contains DHT data or a proposal to collect DHT data. This change created a mechanism for identifying submissions involving DHT‐based remote data acquisition. Following implementation of these fields, the Office of Medical Policy in FDA's Center for Drug Evaluation and Research developed and implemented a systematic process to track, identify, and adjudicate DHT‐related submissions. This analytical report presents a descriptive, aggregated, de‐identified analysis of submissions that were flagged through this operational process from March 2023 through December 2025. Operational inclusion criteria were applied to identify submissions involving DHTs used for remote data acquisition intended to support regulatory decision‐making. Of 2318 flagged submissions, 277 (12%) were confirmed for inclusion in this analysis. Confirmed submissions were predominantly associated with investigational applications (237/277, 86%). When classified by the nature of data collected, submissions reflected a range of remote data acquisition approaches, including patient reported data and sensor‐based measurements, with some submissions combining more than one modality. Electronic diary and electronic patient‐reported outcomes were most common, exclusively appearing in 161 submissions (58%), followed by data captured from general objective sensors in 92 submissions (33%) and continuous glucose monitoring data in 24 submissions (9%). These findings establish a quantitative baseline for DHT‐based remote data acquisition in CDER submissions and the implementation of FDA's corresponding guidance.
- Open Access
- 13 October 2026
- Published by Springer Nature in npj Digital Medicine
Development of a scoring system via an interpretable end-to-end neural network for prognostic stratification of patients with advanced melanoma
We created an interpretable end-to-end neural network (ScoreNet) which can be used to develop an easy-to-implement scoring system for risk stratification. We applied it to data from 2711 patients with advanced melanoma across 5 trials that supported new drug approvals and developed a scoring system for overall survival (OS) and progression-free survival (PFS). The scoring system was then validated in two separate trials. This score is a weighted summation based on eight baseline features (sum of longest diameter for all target lesions, lactate dehydrogenase, systemic immune inflammation index, hemoglobin, ECOG performance status, liver metastasis, pulse rate, and PD-L1 immunohistochemistry status). Using the score, patients can be classified into low, moderate, or high-risk categories of reduced OS and PFS. Risk stratification was demonstrated for patients treated with PD-1/PD-L1 inhibitors, chemotherapy, CTLA-4 inhibitors, BRAF or MEK inhibitors, and their combinations. ScoreNet-based risk stratification can be a new tool for clinical research/practice.
- 13 October 2026
- Published by Springer Nature in npj Digital Medicine
Open AccessDevelopment of a scoring system via an interpretable end-to-end neural network for prognostic stratification of patients with advanced melanoma
We created an interpretable end-to-end neural network (ScoreNet) which can be used to develop an easy-to-implement scoring system for risk stratification. We applied it to data from 2711 patients with advanced melanoma across 5 trials that supported new drug approvals and developed a scoring system for overall survival (OS) and progression-free survival (PFS). The scoring system was then validated in two separate trials. This score is a weighted summation based on eight baseline features (sum of longest diameter for all target lesions, lactate dehydrogenase, systemic immune inflammation index, hemoglobin, ECOG performance status, liver metastasis, pulse rate, and PD-L1 immunohistochemistry status). Using the score, patients can be classified into low, moderate, or high-risk categories of reduced OS and PFS. Risk stratification was demonstrated for patients treated with PD-1/PD-L1 inhibitors, chemotherapy, CTLA-4 inhibitors, BRAF or MEK inhibitors, and their combinations. ScoreNet-based risk stratification can be a new tool for clinical research/practice.
- Open Access
- 7 September 2026
- Published by Wiley in Clinical and Translational Science
- Vol. 19 (9) , e70727
Artificial Intelligence in Model‐Informed Drug Development: Augmentation, Acceleration, and Regulatory Preparation
The convergence of MIDD and AI represents a natural evolution from established quantitative frameworks toward a broader, more powerful computational toolbox that can address persistent MIDD challenges such as high‐dimensional data integration, complex biology, and operational constraints in evidence generation. AI can both enhance traditional MIDD through hybrid approaches and extend MIDD's horizon to include various AI models. Recent publications of guidance demonstrate the agency's preparation on the acceptance of AI in MIDD.
- 7 September 2026
- Published by Wiley in Clinical and Translational Science
- Vol. 19 (9) , e70727
Open AccessArtificial Intelligence in Model‐Informed Drug Development: Augmentation, Acceleration, and Regulatory Preparation
The convergence of MIDD and AI represents a natural evolution from established quantitative frameworks toward a broader, more powerful computational toolbox that can address persistent MIDD challenges such as high‐dimensional data integration, complex biology, and operational constraints in evidence generation. AI can both enhance traditional MIDD through hybrid approaches and extend MIDD's horizon to include various AI models. Recent publications of guidance demonstrate the agency's preparation on the acceptance of AI in MIDD.
- 3 September 2026
- Published by Wiley in Clinical Pharmacology & Therapeutics
AI Cautionary Guide: Pitfalls and Strategies for the Use of Machine Learning in Medical Research
Machine learning ( ML ) is increasingly utilized in medical research. However, underrecognized methodological pitfalls can result in misleading conclusions, which if acted upon can result in patient mismanagement and harm. As a result, we have provided a compilation of common pitfalls and practical safeguard considerations for ML ‐enabled medical research. Examples emphasize the importance of integrating clinical/biologic knowledge and statistical/epidemiologic principles, illustrated with key case examples. Topics include (1) Data drift, which can create artificial trends when coding systems or clinical practices change over time; (2) Feature construction that should be clinically grounded, as models may otherwise learn operational proxies rather than true biology; (3) Accidental data leakage, which can inflate apparent model performance and collapse on real‐world deployment; (4) Competing risks and differential time‐at‐risk that can generate spurious associations in safety analyses if not modeled explicitly; (5) Confounding and collider bias, which can distort findings, particularly in EHR ‐based studies; (6) For rare outcomes, inappropriate evaluation measures can overstate performance and mislead explainability analyses; (7) Finally, pooled multi‐trial analyses require careful handling of the intrinsic heterogeneity to avoid extracting study artifacts rather than generalizable signals. By outlining many of the common potential issues, with lessons learned, we aim to provide the medical and research community with practical insights to help foster trustworthy ML ‐enabled medical research.
- 3 September 2026
- Published by Wiley in Clinical Pharmacology & Therapeutics
AI Cautionary Guide: Pitfalls and Strategies for the Use of Machine Learning in Medical Research
Machine learning ( ML ) is increasingly utilized in medical research. However, underrecognized methodological pitfalls can result in misleading conclusions, which if acted upon can result in patient mismanagement and harm. As a result, we have provided a compilation of common pitfalls and practical safeguard considerations for ML ‐enabled medical research. Examples emphasize the importance of integrating clinical/biologic knowledge and statistical/epidemiologic principles, illustrated with key case examples. Topics include (1) Data drift, which can create artificial trends when coding systems or clinical practices change over time; (2) Feature construction that should be clinically grounded, as models may otherwise learn operational proxies rather than true biology; (3) Accidental data leakage, which can inflate apparent model performance and collapse on real‐world deployment; (4) Competing risks and differential time‐at‐risk that can generate spurious associations in safety analyses if not modeled explicitly; (5) Confounding and collider bias, which can distort findings, particularly in EHR ‐based studies; (6) For rare outcomes, inappropriate evaluation measures can overstate performance and mislead explainability analyses; (7) Finally, pooled multi‐trial analyses require careful handling of the intrinsic heterogeneity to avoid extracting study artifacts rather than generalizable signals. By outlining many of the common potential issues, with lessons learned, we aim to provide the medical and research community with practical insights to help foster trustworthy ML ‐enabled medical research.
- Open Access
- 5 August 2026
- Published by Wiley in Clinical and Translational Science
- Vol. 19 (8) , e70686
Empowering Clinical Development With Disease Progression Modeling: Recommendations From the Clinical Trials Transformation Initiative
Evaluating the benefit–risk profile of a medical product requires comprehensive evidence that integrates information across development stages. Technological advances and broader use of real‐world data are enabling innovative quantitative paradigms characterized by greater efficiency, patient centricity, and sustainability. Despite regulatory recognition of model‐informed drug development, disease progression modeling (DPM) remains underutilized compared with more established pharmacokinetic/pharmacodynamic modeling and simulation approaches. To inform adoption decisions by clinical, regulatory, and portfolio leaders, the Clinical Trials Transformation Initiative (CTTI) developed evidence‐based, cross‐sector recommendations that describe when and how to use DPM in medical product development and how to effectively communicate its impactful implementation across functions. In this article, we (i) define DPM and summarize its unique value in clinical development, (ii) describe CTTI's collaborative process to generate a set of nine DPM recommendations and a practical considerations framework, and (iii) present examples of strategic development questions showing how DPM can answer high‐value questions about indication, population, endpoints, and dose selection. By providing a shared vocabulary and structured questions for decision makers and modelers, these recommendations may lower barriers to DPM implementation, support fit‐for‐purpose use of existing and new models, and enable more efficient, patient‐focused, and sustainable clinical development.
- 5 August 2026
- Published by Wiley in Clinical and Translational Science
- Vol. 19 (8) , e70686
Open AccessEmpowering Clinical Development With Disease Progression Modeling: Recommendations From the Clinical Trials Transformation Initiative
Evaluating the benefit–risk profile of a medical product requires comprehensive evidence that integrates information across development stages. Technological advances and broader use of real‐world data are enabling innovative quantitative paradigms characterized by greater efficiency, patient centricity, and sustainability. Despite regulatory recognition of model‐informed drug development, disease progression modeling (DPM) remains underutilized compared with more established pharmacokinetic/pharmacodynamic modeling and simulation approaches. To inform adoption decisions by clinical, regulatory, and portfolio leaders, the Clinical Trials Transformation Initiative (CTTI) developed evidence‐based, cross‐sector recommendations that describe when and how to use DPM in medical product development and how to effectively communicate its impactful implementation across functions. In this article, we (i) define DPM and summarize its unique value in clinical development, (ii) describe CTTI's collaborative process to generate a set of nine DPM recommendations and a practical considerations framework, and (iii) present examples of strategic development questions showing how DPM can answer high‐value questions about indication, population, endpoints, and dose selection. By providing a shared vocabulary and structured questions for decision makers and modelers, these recommendations may lower barriers to DPM implementation, support fit‐for‐purpose use of existing and new models, and enable more efficient, patient‐focused, and sustainable clinical development.
- 26 November 2025
- Published by Massachusetts Medical Society in NEJM AI
- Vol. 2 (12)
Mitigating Limited Data Challenges to Improve Artificial Intelligence Integration in Rare Disease Drug Development
The Orphan Drug Act defines a rare disease as a condition affecting fewer than 200,000 people in the United States. However, most rare diseases are categorized as ultrarare or hyper-rare, impacting fewer than 100 individuals worldwide. Developing drugs for these conditions involves multiple challenges, such as geographically dispersed and small patient populations, limited natural history data, and poor disease characterization. Issues related to small patient numbers, scarce natural history information, and clinical heterogeneity within rare diseases can be addressed by various strategies, including using artificial intelligence and advanced analytical methods, leveraging detailed individual-level data, and exploring synthetic data generation to overcome the limitations of small datasets. Moreover, establishing centralized databases and promoting public–private partnerships can help build a more comprehensive repository of available data.
- 26 November 2025
- Published by Massachusetts Medical Society in NEJM AI
- Vol. 2 (12)
Mitigating Limited Data Challenges to Improve Artificial Intelligence Integration in Rare Disease Drug Development
The Orphan Drug Act defines a rare disease as a condition affecting fewer than 200,000 people in the United States. However, most rare diseases are categorized as ultrarare or hyper-rare, impacting fewer than 100 individuals worldwide. Developing drugs for these conditions involves multiple challenges, such as geographically dispersed and small patient populations, limited natural history data, and poor disease characterization. Issues related to small patient numbers, scarce natural history information, and clinical heterogeneity within rare diseases can be addressed by various strategies, including using artificial intelligence and advanced analytical methods, leveraging detailed individual-level data, and exploring synthetic data generation to overcome the limitations of small datasets. Moreover, establishing centralized databases and promoting public–private partnerships can help build a more comprehensive repository of available data.
- 26 November 2025
- Published by Massachusetts Medical Society in NEJM AI
- Vol. 2 (12)
Leveraging Artificial Intelligence in Drug and Biological Product Development: An FDA and Clinical Trial Transformation Initiative Workshop Report
Artificial intelligence (AI) holds immense potential to transform drug development by improving the efficiency and accuracy of key processes across the drug product life cycle. However, the scalable adoption of this technology may be influenced by new and unique challenges. The U.S. Food and Drug Administration collaborated with the Clinical Trial Transformation Initiative to organize a public workshop on Artificial Intelligence in Drug and Biological Product Development in August 2024 with medical product sponsors, technology innovators, academicians, and regulators to discuss guiding principles for the use of AI in drug and biological product development in order to realize its transformative potential. This article synthesizes key insights from the workshop and discusses the emerging current need for policy development to enhance the integration of AI in drug and biological product development.
- 26 November 2025
- Published by Massachusetts Medical Society in NEJM AI
- Vol. 2 (12)
Leveraging Artificial Intelligence in Drug and Biological Product Development: An FDA and Clinical Trial Transformation Initiative Workshop Report
Artificial intelligence (AI) holds immense potential to transform drug development by improving the efficiency and accuracy of key processes across the drug product life cycle. However, the scalable adoption of this technology may be influenced by new and unique challenges. The U.S. Food and Drug Administration collaborated with the Clinical Trial Transformation Initiative to organize a public workshop on Artificial Intelligence in Drug and Biological Product Development in August 2024 with medical product sponsors, technology innovators, academicians, and regulators to discuss guiding principles for the use of AI in drug and biological product development in order to realize its transformative potential. This article synthesizes key insights from the workshop and discusses the emerging current need for policy development to enhance the integration of AI in drug and biological product development.
- Open Access
- 3 November 2025
- Published by Wiley in Clinical and Translational Science
- Vol. 18 (11) , e70383
Artificial Intelligence in Clinical and Translational Science : From Bench Insights to Bedside Impact
- 3 November 2025
- Published by Wiley in Clinical and Translational Science
- Vol. 18 (11) , e70383
Open AccessArtificial Intelligence in Clinical and Translational Science : From Bench Insights to Bedside Impact
- Open Access
- 24 October 2025
- Published by The Open Journal in Journal of Open Source Software
- Vol. 10 (114) , 8026
Calzone: A Python package for measuring calibration of probabilistic models for classification
Calzone is a Python package for evaluating the calibration of probabilistic outputs of classifier models. It provides a set of functions for visualizing calibration and computing calibration metrics given a representative dataset with the model's predictions and the true class labels. The metrics provided in Calzone include: expected calibration error (ECE), maximum calibration error (MCE), Hosmer-Lemeshow (HL) test, integrated calibration index (ICI), Spiegelhalter's Z-statistic and Cox's calibration slope/intercept. The package is designed with versatility in mind. Many metrics allow users to adjust binning schemes and choose between top-class or class-wise calculations.
- 24 October 2025
- Published by The Open Journal in Journal of Open Source Software
- Vol. 10 (114) , 8026
Open AccessCalzone: A Python package for measuring calibration of probabilistic models for classification
Calzone is a Python package for evaluating the calibration of probabilistic outputs of classifier models. It provides a set of functions for visualizing calibration and computing calibration metrics given a representative dataset with the model's predictions and the true class labels. The metrics provided in Calzone include: expected calibration error (ECE), maximum calibration error (MCE), Hosmer-Lemeshow (HL) test, integrated calibration index (ICI), Spiegelhalter's Z-statistic and Cox's calibration slope/intercept. The package is designed with versatility in mind. Many metrics allow users to adjust binning schemes and choose between top-class or class-wise calculations.
- Open Access
- 23 October 2025
- Published by Wiley in Clinical and Translational Science
- Vol. 18 (10) , e70349
Out‐of‐Distribution Detection as a Risk‐Control Strategy for Medical Classification Machine Learning Models
Recent advances in artificial intelligence (AI) and machine learning (ML) algorithms have significantly broadened the scope in which these algorithms are deployed. The use of these algorithms in high‐stakes medical contexts requires that the algorithms have appropriate guardrails to ensure their performance. One such guardrail involves using out‐of‐distribution (OOD) detection algorithms, which detect if observations are unlikely to be sampled from the model's training distribution. Since these observations are not observed/rare in the training data, the model is likely to be unreliable when performing on these observations. In the medical context, identifying which patients are OOD may improve the performance of the model by filtering out patients on which the model has not been properly trained or tested. Here, we assess the performance of state‐of‐the‐art OOD detection algorithms on three medical datasets of image, transcriptomics, and time series modalities, respectively. Using a simulated training‐deployment scenario, we find that several OOD detectors consistently identify patients on which the model performs worse. Further, several OOD detectors identified patient subsets that were underrepresented in the training data, prompting further investigation. We present that the use of OOD detection methods could help mitigate model risk when deploying medical AI in the real world.
- 23 October 2025
- Published by Wiley in Clinical and Translational Science
- Vol. 18 (10) , e70349
Open AccessOut‐of‐Distribution Detection as a Risk‐Control Strategy for Medical Classification Machine Learning Models
Recent advances in artificial intelligence (AI) and machine learning (ML) algorithms have significantly broadened the scope in which these algorithms are deployed. The use of these algorithms in high‐stakes medical contexts requires that the algorithms have appropriate guardrails to ensure their performance. One such guardrail involves using out‐of‐distribution (OOD) detection algorithms, which detect if observations are unlikely to be sampled from the model's training distribution. Since these observations are not observed/rare in the training data, the model is likely to be unreliable when performing on these observations. In the medical context, identifying which patients are OOD may improve the performance of the model by filtering out patients on which the model has not been properly trained or tested. Here, we assess the performance of state‐of‐the‐art OOD detection algorithms on three medical datasets of image, transcriptomics, and time series modalities, respectively. Using a simulated training‐deployment scenario, we find that several OOD detectors consistently identify patients on which the model performs worse. Further, several OOD detectors identified patient subsets that were underrepresented in the training data, prompting further investigation. We present that the use of OOD detection methods could help mitigate model risk when deploying medical AI in the real world.