Addressing Power of Equivalence Tests for Statistical Fairness Assessments of Machine Learning Models in Healthcare
Open Access
Abstract
Background/Objectives: Machine learning and artificial intelligence (AI) have substantial potential to change the practice of oncology through individualized patient care; however, it is important to ensure that any benefits are realized equitably. There are several examples of AI tools exhibiting bias towards traditionally under-resourced or marginalized groups. Proper validation that these models perform well across populations requires rigorous statistical design and interpretation. One component that is currently lacking in fairness assessments is a clear, prespecified definition of the evidence required to conclude that an AI tool is fair. Methods: To address this issue, we embed the notion of fairness assessment of an AI model within the statistical equivalence testing framework. This approach provides an a priori definition of fairness and an explicit decision rule for concluding, with a high degree of confidence, that model performance falls within a prespecified margin. The framework also supplies power and sample size calculations to support the design phase of a fairness assessment, which we develop for the case of modeling a binary outcome and a binary classification. As a demonstration, we use a “four-fifths” rule of thumb to set equivalence margins but emphasize the necessity of clinical and ethical context for choosing the margins in practice. Results: Using two published oncology models, we show that a realistic sample size can be adequate for some performance metrics and inadequate for others. We also address power and sample size considerations for asymmetric association measures and impossibility results for fairness. Conclusions: Assessments of AI performance fairness should be designed with intention such that target metrics and equivalence margins are specified in advance with ample sample size to conclude fairness.Keywords
Funding Information
- National Heart, Lung, and Blood Institute (R01HL171508)
- National Center for Advancing Translational Sciences (UL1TR002377)
This publication has 61 references indexed in Scilit:
- Ancestry-Associated Performance Variability of Open-Source AI Models for EGFR Prediction in Lung CancerJAMA Oncology, 2026
- Bringing practical statistical science to AI and predictive model fairness testingAI and Ethics, 2024
- Risk Model–Based Lung Cancer Screening and Racial and Ethnic Disparities in the USJAMA Oncology, 2023
- Racial and Ethnic Bias in Risk Prediction Models for Colorectal Cancer Recurrence When Race and Ethnicity Are Omitted as PredictorsJAMA Network Open, 2023
- Risk Prediction Model Versus United States Preventive Services Task Force Lung Cancer Screening Eligibility Criteria: Reducing Race DisparitiesJournal of Thoracic Oncology, 2020
- Advancing quantitative intersectionality research methods: Intracategorical and intercategorical approaches to shared and differential constructsSocial Science & Medicine, 2019
- Understanding Equivalence and Noninferiority TestingJournal of General Internal Medicine, 2010
- Model validation using equivalence testsEcological Modelling, 2004
- A simple method for converting an odds ratio to effect size for use in meta-analysisStatistics in Medicine, 2000
- A comparison of the Two One-Sided Tests Procedure and the Power Approach for assessing the equivalence of average bioavailabilityJournal of Pharmacokinetics and Biopharmaceutics, 1987