Disi Ji
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Disi Ji has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Padhraic Smyth
Department of Computer Science University of California Irvine
3 shared publications - Eric Nalisnick
Department of Computer Science University of California Irvine
2 shared publications - Mark Steyvers
University of California, Irvine, Irvine, USA
2 shared publications - Padhraic Smyth
Department of Statistics
1 shared publication - Richard H. Scheuermann
Division of Intramural Research, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
1 shared publication - Robert L. Logan Iv
Department of Computer Science University of California Irvine
1 shared publication
Latest Publications
- Open Access
Active Bayesian Assessment for Black-Box Classifiers
Recent advances in machine learning have led to increased deployment of black-box classifiers across a wide variety of applications. In many such situations there is a critical need to both reliably assess the performance of these pre-trained models and to perform this assessment in a label-efficient manner (given that labels may be scarce and costly to collect). In this paper, we introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We begin by developing inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling, enabling better performance assessment with fewer labels. We demonstrate significant gains from our proposed active Bayesian approach via a series of systematic empirical experiments assessing the performance of modern neural classifiers (e.g., ResNet and BERT) on several standard image and text classification datasets.
Open AccessActive Bayesian Assessment for Black-Box Classifiers
Recent advances in machine learning have led to increased deployment of black-box classifiers across a wide variety of applications. In many such situations there is a critical need to both reliably assess the performance of these pre-trained models and to perform this assessment in a label-efficient manner (given that labels may be scarce and costly to collect). In this paper, we introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We begin by developing inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling, enabling better performance assessment with fewer labels. We demonstrate significant gains from our proposed active Bayesian approach via a series of systematic empirical experiments assessing the performance of modern neural classifiers (e.g., ResNet and BERT) on several standard image and text classification datasets.
- Open Access
Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference
We investigate the problem of reliably assessing group fairness when labeled examples are few but unlabeled examples are plentiful. We propose a general Bayesian framework that can augment labeled data with unlabeled data to produce more accurate and lower-variance estimates compared to methods based on labeled data alone. Our approach estimates calibrated scores for unlabeled examples in each group using a hierarchical latent variable model conditioned on labeled examples. This in turn allows for inference of posterior distributions with associated notions of uncertainty for a variety of group fairness metrics. We demonstrate that our approach leads to significant and consistent reductions in estimation error across multiple well-known fairness datasets, sensitive attributes, and predictive models. The results show the benefits of using both unlabeled data and Bayesian inference in terms of assessing whether a prediction model is fair or not.
Open AccessCan I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference
We investigate the problem of reliably assessing group fairness when labeled examples are few but unlabeled examples are plentiful. We propose a general Bayesian framework that can augment labeled data with unlabeled data to produce more accurate and lower-variance estimates compared to methods based on labeled data alone. Our approach estimates calibrated scores for unlabeled examples in each group using a hierarchical latent variable model conditioned on labeled examples. This in turn allows for inference of posterior distributions with associated notions of uncertainty for a variety of group fairness metrics. We demonstrate that our approach leads to significant and consistent reductions in estimation error across multiple well-known fairness datasets, sensitive attributes, and predictive models. The results show the benefits of using both unlabeled data and Bayesian inference in terms of assessing whether a prediction model is fair or not.
- Open Access
Bayesian Trees for Automated Cytometry Data Analysis
Cytometry is an important single cell analysis technology in furthering our understanding of cellular biological processes and in supporting clinical diagnoses across a variety hematological and immunological conditions. Current data analysis workflows for cytometry data rely on a manual process called gating to classify cells into canonical types. This dependence on human annotation significantly limits the rate, reproducibility, and scope of cytometry’s use in both biological research and clinical practice. We develop a novel Bayesian approach for automated gating that classifies cells into different types by combining cell-level marker measurements with an informative prior. The Bayesian approach allows for the incorporation of biologically-meaningful prior information that captures the domain expertise of human experts. The inference algorithm results in a hierarchically-structured classification of individual cells in a manner that mimics the tree-structured recursive process of manual gating, making the results readily interpretable. The approach can be extended in a natural fashion to handle data from multiple different samples by the incorporation of random effects in the Bayesian model. The proposed approach is evaluated using mass cytometry data, on the problems of unsupervised cell classification and supervised clinical diagnosis, illustrating the benefits of both incorporating prior knowledge and sharing information across multiple samples.
Open AccessBayesian Trees for Automated Cytometry Data Analysis
Cytometry is an important single cell analysis technology in furthering our understanding of cellular biological processes and in supporting clinical diagnoses across a variety hematological and immunological conditions. Current data analysis workflows for cytometry data rely on a manual process called gating to classify cells into canonical types. This dependence on human annotation significantly limits the rate, reproducibility, and scope of cytometry’s use in both biological research and clinical practice. We develop a novel Bayesian approach for automated gating that classifies cells into different types by combining cell-level marker measurements with an informative prior. The Bayesian approach allows for the incorporation of biologically-meaningful prior information that captures the domain expertise of human experts. The inference algorithm results in a hierarchically-structured classification of individual cells in a manner that mimics the tree-structured recursive process of manual gating, making the results readily interpretable. The approach can be extended in a natural fashion to handle data from multiple different samples by the incorporation of random effects in the Bayesian model. The proposed approach is evaluated using mass cytometry data, on the problems of unsupervised cell classification and supervised clinical diagnosis, illustrating the benefits of both incorporating prior knowledge and sharing information across multiple samples.
- Open Access
Mondrian Processes for Flow Cytometry Analysis
Analysis of flow cytometry data is an essential tool for clinical diagnosis of hematological and immunological conditions. Current clinical workflows rely on a manual process called gating to classify cells into their canonical types. This dependence on human annotation limits the rate, reproducibility, and complexity of flow cytometry analysis. In this paper, we propose using Mondrian processes to perform automated gating by incorporating prior information of the kind used by gating technicians. The method segments cells into types via Bayesian nonparametric trees. Examining the posterior over trees allows for interpretable visualizations and uncertainty quantification - two vital qualities for implementation in clinical practice.
Open AccessMondrian Processes for Flow Cytometry Analysis
Analysis of flow cytometry data is an essential tool for clinical diagnosis of hematological and immunological conditions. Current clinical workflows rely on a manual process called gating to classify cells into their canonical types. This dependence on human annotation limits the rate, reproducibility, and complexity of flow cytometry analysis. In this paper, we propose using Mondrian processes to perform automated gating by incorporating prior information of the kind used by gating technicians. The method segments cells into types via Bayesian nonparametric trees. Examining the posterior over trees allows for interpretable visualizations and uncertainty quantification - two vital qualities for implementation in clinical practice.