The Label Complexity of Active Learning from Observational Data
Preprint
Abstract
Counterfactual learning from observational data involves learning a classifier on an entire population based on data that is observed conditioned on a selection policy. This work considers this problem in an active setting, where the learner additionally has access to unlabeled examples and can choose to get a subset of these labeled by an oracle. Prior work on this problem uses disagreement-based active learning, along with an importance weighted loss estimator to account for counterfactuals, which leads to a high label complexity. We show how to instead incorporate a more efficient counterfactual risk minimizer into the active learning algorithm. This requires us to modify both the counterfactual risk to make it amenable to active learning, as well as the active learning process to make it amenable to the risk. We provably demonstrate that the result of this is an algorithm which is statistically consistent as well as more label-efficient than prior work.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Effective Evaluation Using Logged Bandit Feedback from Multiple Loggers2017
- Theory of Disagreement-Based Active LearningFoundations and Trends in Machine Learning, 2014
- A bound on the label complexity of agnostic active learning2007
- Simple Approximations of the Error Function Q(x) for Communications ApplicationsIEEE Transactions on Communications, 1979