Tyler H. McCormick
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Tyler H. McCormick has a total of 156 co-authors. Here are the top co-authors by number of shared publications:
- Samuel J. Clark
Department of Sociology, The Ohio State University
7 shared publications - Stephen Salerno
Public Health Sciences Biostatistics Fred Hutchinson Cancer Center Seattle WA
6 shared publications - Tian Zheng
Department of Statistics Columbia University New York NY USA
6 shared publications - Zehang Richard Li
Department of Statistics, University of California Santa Cruz
6 shared publications Kentaro Hoffman
5 shared publications- Awan Afiaz
Public Health Sciences Biostatistics Fred Hutchinson Cancer Center Biostatistics University of Washington Seattle WA
4 shared publications
Latest Publications
- Open Access
Multimodal Artificial Intelligence for Cause-of-Death Assignment From Verbal Autopsy Records: A National Benchmark Study in South Africa
Background: Timely cause-of-death (COD) information guides health policy, yet many deaths in low- and middle-income countries are not medically certified. In Africa, fewer than 10% of deaths are. Verbal autopsy (VA) yields a structured symptom questionnaire, and a free-text narrative. Mortality surveillance increasingly records the narrative alongside the questionnaire, but the algorithms used to produce VA-based statistics were designed for the structured responses only, so narratives inform physician COD assignment but do not contribute to algorithmic COD assignment. We evaluated whether supervised models could use both modalities to improve COD assignment. Methods: We analyzed 4,999 VA records for deaths at ages 12 years or older from a national sample of registered deaths in South Africa (RSA), using physician-assigned CODs as the reference standard. We trained unimodal and multimodal pipelines on the labelled records: pretrained language models (PLMs) fine-tuned on narratives alone, machine learning (ML) models on structured responses alone, and the two combined at the data, feature and decision levels. We also prompted a locally deployed generative large language model (LLM) on the same unimodal and multimodal inputs, without parameter updates. All were compared with InSilicoVA, the operational comparator representing current automated practice. We evaluated individual-level assignment, population-level agreement, calibration, and generalization across provinces in RSA, reporting these metrics rather than ranking pipelines on any one of them. Results: Every supervised pipeline improved on InSilicoVA by 8.5–15.9 percentage points in top-1 accuracy and 3.5–7.1 in cause-specific mortality fraction (CSMF) accuracy. The narrative and structured pipelines performed similarly in aggregate but differed by cause: 63.9% of deaths received identical predictions, 11.6% were correct only from the narrative pipeline and 13.6% only from the questionnaire pipeline, with the narrative pipeline more accurate for noncommunicable causes and the questionnaire pipeline for communicable causes. Against the unimodal approaches, the decision-fusion pipelines added 4.2 to 5.4 percentage points in top-1 accuracy, concentrated in the most common causes, whereas feature fusion with cross-modal attention improved the less common causes instead, adding 6.6 (1.7–11.5) points in balanced accuracy and 5.5 (0.6–10.3) in macro F1. Feature fusion reached the highest CSMF accuracy (95.7%). Paired differences from the better unimodal representative ranged from − 0.2 to +3.4 points, with all intervals including zero. The LLM exceeded InSilicoVA on individual-level metrics and reached the highest balanced accuracy of any pipeline (60.6%), while the supervised pipelines retained higher CSMF accuracy. Supervised pipelines had lower expected calibration error than InSilicoVA. Supervised gains held across all nine held-out provinces, including every top-1 comparison and nearly every macro F1 and CSMF comparison. Conclusions: PLMs using narratives alone substantially improved on the current best practice and matched ML pipelines using structured responses, while multiple multimodal strategies produced further gains in individual COD assignment. The zero-shot LLM showed particular strength on rare causes, whereas supervised pipelines better recovered the population cause distribution. These findings support incorporating narratives as a core input to automated VA systems and evaluating representative fusion strategies for the intended output. External validation across settings and age groups will support broader implementation.
Open AccessMultimodal Artificial Intelligence for Cause-of-Death Assignment From Verbal Autopsy Records: A National Benchmark Study in South Africa
Background: Timely cause-of-death (COD) information guides health policy, yet many deaths in low- and middle-income countries are not medically certified. In Africa, fewer than 10% of deaths are. Verbal autopsy (VA) yields a structured symptom questionnaire, and a free-text narrative. Mortality surveillance increasingly records the narrative alongside the questionnaire, but the algorithms used to produce VA-based statistics were designed for the structured responses only, so narratives inform physician COD assignment but do not contribute to algorithmic COD assignment. We evaluated whether supervised models could use both modalities to improve COD assignment. Methods: We analyzed 4,999 VA records for deaths at ages 12 years or older from a national sample of registered deaths in South Africa (RSA), using physician-assigned CODs as the reference standard. We trained unimodal and multimodal pipelines on the labelled records: pretrained language models (PLMs) fine-tuned on narratives alone, machine learning (ML) models on structured responses alone, and the two combined at the data, feature and decision levels. We also prompted a locally deployed generative large language model (LLM) on the same unimodal and multimodal inputs, without parameter updates. All were compared with InSilicoVA, the operational comparator representing current automated practice. We evaluated individual-level assignment, population-level agreement, calibration, and generalization across provinces in RSA, reporting these metrics rather than ranking pipelines on any one of them. Results: Every supervised pipeline improved on InSilicoVA by 8.5–15.9 percentage points in top-1 accuracy and 3.5–7.1 in cause-specific mortality fraction (CSMF) accuracy. The narrative and structured pipelines performed similarly in aggregate but differed by cause: 63.9% of deaths received identical predictions, 11.6% were correct only from the narrative pipeline and 13.6% only from the questionnaire pipeline, with the narrative pipeline more accurate for noncommunicable causes and the questionnaire pipeline for communicable causes. Against the unimodal approaches, the decision-fusion pipelines added 4.2 to 5.4 percentage points in top-1 accuracy, concentrated in the most common causes, whereas feature fusion with cross-modal attention improved the less common causes instead, adding 6.6 (1.7–11.5) points in balanced accuracy and 5.5 (0.6–10.3) in macro F1. Feature fusion reached the highest CSMF accuracy (95.7%). Paired differences from the better unimodal representative ranged from − 0.2 to +3.4 points, with all intervals including zero. The LLM exceeded InSilicoVA on individual-level metrics and reached the highest balanced accuracy of any pipeline (60.6%), while the supervised pipelines retained higher CSMF accuracy. Supervised pipelines had lower expected calibration error than InSilicoVA. Supervised gains held across all nine held-out provinces, including every top-1 comparison and nearly every macro F1 and CSMF comparison. Conclusions: PLMs using narratives alone substantially improved on the current best practice and matched ML pipelines using structured responses, while multiple multimodal strategies produced further gains in individual COD assignment. The zero-shot LLM showed particular strength on rare causes, whereas supervised pipelines better recovered the population cause distribution. These findings support incorporating narratives as a core input to automated VA systems and evaluating representative fusion strategies for the intended output. External validation across settings and age groups will support broader implementation.
- Open Access
A Multimodal Benchmark for Evaluating Cause-of-Death Inference Using Child Health and Mortality Data
A bs t r act: Accurately attributing causes of death is vital for global health, yet fewer than 5% of deaths in resource-constrained regions are medically certified. To assign causes to these unlabeled deaths at scale, practitioners traditionally rely on verbal autopsy, using supervised statistical models to classify based on structured survey data. However, modern mortality surveillance increasingly collects rich, unstructured multimodal data, such as free-text caregiver narratives and postmortem diagnostics, which traditional supervised statistical models struggle to seamlessly integrate. In this paper, we present a comprehensive, multimodal benchmark for cause-of-death classification using data from the Child Health and Mortality Prevention Surveillance (CHAMPS) network, a unique surveillance platform spanning nine countries across South Asia and Sub-Saharan Africa. Using this dataset, we introduce an evaluation framework designed to rigorously assess diagnostic reasoning, moving beyond traditional metrics that fail to capture complex clinical realities. We demonstrate the utility of this benchmark by evaluating zero-shot large language models against supervised baselines across various data modalities. Our results reveal distinct differences in how these modeling approaches synthesize unstructured medical evidence. This benchmark provide a rigorously defined resource for assessing clinical reasoning in next-generation mortality surveillance.
Open AccessA Multimodal Benchmark for Evaluating Cause-of-Death Inference Using Child Health and Mortality Data
A bs t r act: Accurately attributing causes of death is vital for global health, yet fewer than 5% of deaths in resource-constrained regions are medically certified. To assign causes to these unlabeled deaths at scale, practitioners traditionally rely on verbal autopsy, using supervised statistical models to classify based on structured survey data. However, modern mortality surveillance increasingly collects rich, unstructured multimodal data, such as free-text caregiver narratives and postmortem diagnostics, which traditional supervised statistical models struggle to seamlessly integrate. In this paper, we present a comprehensive, multimodal benchmark for cause-of-death classification using data from the Child Health and Mortality Prevention Surveillance (CHAMPS) network, a unique surveillance platform spanning nine countries across South Asia and Sub-Saharan Africa. Using this dataset, we introduce an evaluation framework designed to rigorously assess diagnostic reasoning, moving beyond traditional metrics that fail to capture complex clinical realities. We demonstrate the utility of this benchmark by evaluating zero-shot large language models against supervised baselines across various data modalities. Our results reveal distinct differences in how these modeling approaches synthesize unstructured medical evidence. This benchmark provide a rigorously defined resource for assessing clinical reasoning in next-generation mortality surveillance.
- Open Access
- 12 June 2026
- Published by Elsevier in SSRN Electronic Journal
From Census to Sensors: The Bureaucratic Regime and the Data Revolution
We examine a profound transformation in the relationship between statistical knowledge and state governance through the lens of two "data regimes." In the earlier bureaucratic regime, government agencies produced expensive, theory-driven data through representative surveys and official reporting mechanisms. These techniques are no longer hegemonic. In the emerging brokerage regime, data production is cheap, decentralized, and dominated by private actors. Passive collection of digital traces is displacing representative sampling; algorithmic pattern detection is supplanting theory-guided analysis; and sensitive administrative data is being recast as raw material for AI systems. In the process, a new, paradoxical, political rationality emerges. The private sector encroaches on critical public functions while supplying the state with precise "truths" about people and collective entities that enhance its capacity to manage them.
- 12 June 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessFrom Census to Sensors: The Bureaucratic Regime and the Data Revolution
We examine a profound transformation in the relationship between statistical knowledge and state governance through the lens of two "data regimes." In the earlier bureaucratic regime, government agencies produced expensive, theory-driven data through representative surveys and official reporting mechanisms. These techniques are no longer hegemonic. In the emerging brokerage regime, data production is cheap, decentralized, and dominated by private actors. Passive collection of digital traces is displacing representative sampling; algorithmic pattern detection is supplanting theory-guided analysis; and sensitive administrative data is being recast as raw material for AI systems. In the process, a new, paradoxical, political rationality emerges. The private sector encroaches on critical public functions while supplying the state with precise "truths" about people and collective entities that enhance its capacity to manage them.
- Open Access
Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data
As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g., rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as substitutes for missing or unobserved data. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to drawing inference with predicted data (IPD) and show that high predictive accuracy does not guarantee valid downstream inference. We show that all such failures reduce to statistical notions of (i) bias, when predictions systematically shift the estimand or distort relationships among variables, and (ii) variance, when uncertainty from the prediction model and the intrinsic variability of the true data are ignored. We then review recent methods for conducting IPD and discuss how this framework is deeply rooted in classical statistical theory. We then comment on some open questions and interesting avenues for future work in this area, and end with some comments on how to use predicted data in scientific studies that is both transparent and statistically principled.
Open AccessDo We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data
As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g., rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as substitutes for missing or unobserved data. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to drawing inference with predicted data (IPD) and show that high predictive accuracy does not guarantee valid downstream inference. We show that all such failures reduce to statistical notions of (i) bias, when predictions systematically shift the estimand or distort relationships among variables, and (ii) variance, when uncertainty from the prediction model and the intrinsic variability of the true data are ignored. We then review recent methods for conducting IPD and discuss how this framework is deeply rooted in classical statistical theory. We then comment on some open questions and interesting avenues for future work in this area, and end with some comments on how to use predicted data in scientific studies that is both transparent and statistically principled.
- Open Access
LAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination
Verbal autopsy (VA) is a critical tool for estimating causes of death in resource-limited settings where medical certification is unavailable. This study presents LA-VA, a proof-of-concept pipeline that combines Large Language Models (LLMs) with traditional algorithmic approaches and embedding-based classification for improved cause-of-death prediction. Using the Population Health Metrics Research Consortium (PHMRC) dataset across three age categories (Adult: 7,580; Child: 1,960; Neonate: 2,438), we evaluate multiple approaches: GPT-5 predictions, LCVA baseline, text embeddings, and meta-learner ensembles. Our results demonstrate that GPT-5 achieves the highest individual performance with average test site accuracies of 48.6% (Adult), 50.5% (Child), and 53.5% (Neonate), outperforming traditional statistical machine learning baselines by 5-10%. Our findings suggest that simple off-the-shelf LLM-assisted approaches could substantially improve verbal autopsy accuracy, with important implications for global health surveillance in low-resource settings.
Open AccessLAVA: Language Model Assisted Verbal Autopsy for Cause-of-Death Determination
Verbal autopsy (VA) is a critical tool for estimating causes of death in resource-limited settings where medical certification is unavailable. This study presents LA-VA, a proof-of-concept pipeline that combines Large Language Models (LLMs) with traditional algorithmic approaches and embedding-based classification for improved cause-of-death prediction. Using the Population Health Metrics Research Consortium (PHMRC) dataset across three age categories (Adult: 7,580; Child: 1,960; Neonate: 2,438), we evaluate multiple approaches: GPT-5 predictions, LCVA baseline, text embeddings, and meta-learner ensembles. Our results demonstrate that GPT-5 achieves the highest individual performance with average test site accuracies of 48.6% (Adult), 50.5% (Child), and 53.5% (Neonate), outperforming traditional statistical machine learning baselines by 5-10%. Our findings suggest that simple off-the-shelf LLM-assisted approaches could substantially improve verbal autopsy accuracy, with important implications for global health surveillance in low-resource settings.
- Open Access
- 4 September 2025
- Published by Cambridge University Press (CUP) in Network Science
- Vol. 13, e11
Spectral goodness-of-fit tests for complete and partial network data
Networks describe complex relationships between individual actors. In this work, we address the question of how to determine whether a parametric model, such as a stochastic block model or latent space model, fits a data set well, and will extrapolate to similar data. We use recent results in random matrix theory to derive a general goodness-of-fit (GoF) test for dyadic data. We show that our method, when applied to a specific model of interest, provides a straightforward, computationally fast way of selecting parameters in a number of commonly used network models. For example, we show how to select the dimension of the latent space in latent space models. Unlike other network GoF methods, our general approach does not require simulating from a candidate parametric model, which can be cumbersome with large graphs, and eliminates the need to choose a particular set of statistics on the graph for comparison. It also allows us to perform GoF tests on partial network data, such as Aggregated Relational Data. We show with simulations that our method performs well in many situations of interest. We analyze several empirically relevant networks and show that our method leads to improved community detection algorithms.
- 4 September 2025
- Published by Cambridge University Press (CUP) in Network Science
- Vol. 13, e11
Open AccessSpectral goodness-of-fit tests for complete and partial network data
Networks describe complex relationships between individual actors. In this work, we address the question of how to determine whether a parametric model, such as a stochastic block model or latent space model, fits a data set well, and will extrapolate to similar data. We use recent results in random matrix theory to derive a general goodness-of-fit (GoF) test for dyadic data. We show that our method, when applied to a specific model of interest, provides a straightforward, computationally fast way of selecting parameters in a number of commonly used network models. For example, we show how to select the dimension of the latent space in latent space models. Unlike other network GoF methods, our general approach does not require simulating from a candidate parametric model, which can be cumbersome with large graphs, and eliminates the need to choose a particular set of statistics on the graph for comparison. It also allows us to perform GoF tests on partial network data, such as Aggregated Relational Data. We show with simulations that our method performs well in many situations of interest. We analyze several empirically relevant networks and show that our method leads to improved community detection algorithms.
- 7 August 2025
- Published by SAGE Publications in Sociological Methods & Research
- Vol. 55 (4) , 1389-1412
Estimating and Correcting Degree Ratio Bias in the Network Scale-up Method
The Network Scale-up Method (NSUM) uses social networks and answers to “How many X’s do you know?” questions to estimate sizes of groups excluded by standard surveys. This paper addresses the bias caused by varying average social network sizes across populations, commonly referred to as the degree ratio bias. This bias likely exists for all populations, resulting in either positively or negatively biased population size estimates. We show how the degree ratio affects size estimates and provide a method to estimate degree ratios without collecting additional data. We demonstrate that our adjustment procedure improves the accuracy of NSUM size estimates using simulations and data from two data sources.
- 7 August 2025
- Published by SAGE Publications in Sociological Methods & Research
- Vol. 55 (4) , 1389-1412
Estimating and Correcting Degree Ratio Bias in the Network Scale-up Method
The Network Scale-up Method (NSUM) uses social networks and answers to “How many X’s do you know?” questions to estimate sizes of groups excluded by standard surveys. This paper addresses the bias caused by varying average social network sizes across populations, commonly referred to as the degree ratio bias. This bias likely exists for all populations, resulting in either positively or negatively biased population size estimates. We show how the degree ratio affects size estimates and provide a method to estimate degree ratios without collecting additional data. We demonstrate that our adjustment procedure improves the accuracy of NSUM size estimates using simulations and data from two data sources.
- Open Access
A Moment-Based Generalization to Post-Prediction Inference
Artificial intelligence (AI) and machine learning (ML) are increasingly used to generate data for downstream analyses, yet naively treating these predictions as true observations can lead to biased results and incorrect inference. Wang et al. (2020) proposed a method, post-prediction inference, which calibrates inference by modeling the relationship between AI/ML-predicted and observed outcomes in a small, gold-standard sample. Since then, several methods have been developed for inference with predicted data. We revisit Wang et al. in light of these recent developments. We reflect on their assumptions and offer a simple extension of their method which relaxes these assumptions. Our extension (1) yields unbiased point estimates under standard conditions and (2) incorporates a simple scaling factor to preserve calibration variability. In extensive simulations, we show that our method maintains nominal Type I error rates, reduces bias, and achieves proper coverage.
Open AccessA Moment-Based Generalization to Post-Prediction Inference
Artificial intelligence (AI) and machine learning (ML) are increasingly used to generate data for downstream analyses, yet naively treating these predictions as true observations can lead to biased results and incorrect inference. Wang et al. (2020) proposed a method, post-prediction inference, which calibrates inference by modeling the relationship between AI/ML-predicted and observed outcomes in a small, gold-standard sample. Since then, several methods have been developed for inference with predicted data. We revisit Wang et al. in light of these recent developments. We reflect on their assumptions and offer a simple extension of their method which relaxes these assumptions. Our extension (1) yields unbiased point estimates under standard conditions and (2) incorporates a simple scaling factor to preserve calibration variability. In extensive simulations, we show that our method maintains nominal Type I error rates, reduces bias, and achieves proper coverage.
- Open Access
How to measure obesity in public health research? Problems with using BMI for population inference
Background Though viewed as problematic for measuring individual-level adiposity, Body Mass Index (BMI) is often considered “good enough” for population inference and epidemiological research. However, we demonstrate that BMI produces statistically invalid population-level estimates of associations between key demographic risk factors (e.g., self-reported sex, race, age) and obesity when compared to more direct adiposity measurements. Further, we demonstrate how novel statistical calibration techniques can enable more valid population inference using widely available BMI data alongside a limited subset of “gold standard” measurements. Methods Using National Health and Nutrition Examination Survey data (2011-2023), we compare associations, broken down by demographic groups, across three different purported adiposity measures: BMI, Waist Circumference (WC), and whole-body total fat percentage from Dual-energy X-ray absorptiometry (DXA) scans. We then apply a statistical procedure for conducting inference on predicted data to calibrate BMI-based prevalence estimates toward the “gold standard” DXA-based measurements, allowing for valid population inference even for time periods where only BMI data are available. Findings BMI-measured adiposity yields substantially different – and often contradictory – conclusions about the association between obesity status and lifestyle factors compared to the more direct, DXA-based measurements. Most concerning, the directions and magnitudes of the associations between racial groups may differ depending on whether BMI or DXA-based measurements are used. Similarly, self-reported sex-based differences in obesity prevalence show opposite patterns across measurement types. Our validation results confirm that our calibration method overcomes this challenge and successfully approximates DXA-based associations using primarily BMI-based measurements. Interpretation Our study provides empirical evidence that uncorrected BMI-based inference leads to invalid population-level estimates about the associations between obesity status and key predictors. The statistical calibration approach we present offers a practical solution for obesity researchers who must rely on BMI or similar anthropometric measures due to cost or data availability constraints, enabling more valid population inference without requiring comprehensive “gold standard” adiposity measurements.
Open AccessHow to measure obesity in public health research? Problems with using BMI for population inference
Background Though viewed as problematic for measuring individual-level adiposity, Body Mass Index (BMI) is often considered “good enough” for population inference and epidemiological research. However, we demonstrate that BMI produces statistically invalid population-level estimates of associations between key demographic risk factors (e.g., self-reported sex, race, age) and obesity when compared to more direct adiposity measurements. Further, we demonstrate how novel statistical calibration techniques can enable more valid population inference using widely available BMI data alongside a limited subset of “gold standard” measurements. Methods Using National Health and Nutrition Examination Survey data (2011-2023), we compare associations, broken down by demographic groups, across three different purported adiposity measures: BMI, Waist Circumference (WC), and whole-body total fat percentage from Dual-energy X-ray absorptiometry (DXA) scans. We then apply a statistical procedure for conducting inference on predicted data to calibrate BMI-based prevalence estimates toward the “gold standard” DXA-based measurements, allowing for valid population inference even for time periods where only BMI data are available. Findings BMI-measured adiposity yields substantially different – and often contradictory – conclusions about the association between obesity status and lifestyle factors compared to the more direct, DXA-based measurements. Most concerning, the directions and magnitudes of the associations between racial groups may differ depending on whether BMI or DXA-based measurements are used. Similarly, self-reported sex-based differences in obesity prevalence show opposite patterns across measurement types. Our validation results confirm that our calibration method overcomes this challenge and successfully approximates DXA-based associations using primarily BMI-based measurements. Interpretation Our study provides empirical evidence that uncorrected BMI-based inference leads to invalid population-level estimates about the associations between obesity status and key predictors. The statistical calibration approach we present offers a practical solution for obesity researchers who must rely on BMI or similar anthropometric measures due to cost or data availability constraints, enabling more valid population inference without requiring comprehensive “gold standard” adiposity measurements.
- Open Access
Unique Rashomon Sets for Robust Active Learning
Collecting labeled data for machine learning models is often expensive and time-consuming. Active learning addresses this challenge by selectively labeling the most informative observations, but when initial labeled data is limited, it becomes difficult to distinguish genuinely informative points from those appearing uncertain primarily due to noise. Ensemble methods like random forests are a powerful approach to quantifying this uncertainty but do so by aggregating all models indiscriminately. This includes poor performing models and redundant models, a problem that worsens in the presence of noisy data. We introduce UNique Rashomon Ensembled Active Learning (UNREAL), which selectively ensembles only distinct models from the Rashomon set, which is the set of nearly optimal models. Restricting ensemble membership to high-performing models with different explanations helps distinguish genuine uncertainty from noise-induced variation. We show that UNREAL achieves faster theoretical convergence rates than traditional active learning approaches and demonstrates empirical improvements of up to 20% in predictive accuracy across five benchmark datasets, while simultaneously enhancing model interpretability.
Open AccessUnique Rashomon Sets for Robust Active Learning
Collecting labeled data for machine learning models is often expensive and time-consuming. Active learning addresses this challenge by selectively labeling the most informative observations, but when initial labeled data is limited, it becomes difficult to distinguish genuinely informative points from those appearing uncertain primarily due to noise. Ensemble methods like random forests are a powerful approach to quantifying this uncertainty but do so by aggregating all models indiscriminately. This includes poor performing models and redundant models, a problem that worsens in the presence of noisy data. We introduce UNique Rashomon Ensembled Active Learning (UNREAL), which selectively ensembles only distinct models from the Rashomon set, which is the set of nearly optimal models. Restricting ensemble membership to high-performing models with different explanations helps distinguish genuine uncertainty from noise-induced variation. We show that UNREAL achieves faster theoretical convergence rates than traditional active learning approaches and demonstrates empirical improvements of up to 20% in predictive accuracy across five benchmark datasets, while simultaneously enhancing model interpretability.