ARAaron Roth

Aaron Roth

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Aaron Roth has a total of 42 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Perceptions of Fairness in Algorithmic Decision-Making

    by Bethany Hsiao, Hamsa Bastani, Michael Kearns, Aaron Roth, Duncan Watts

    In algorithmic decision-making, it is often mathematically impossible to satisfy competing definitions of fairness simultaneously. This impossibility makes assessing stakeholders' fairness preferences a challenging but crucial exercise. In this paper, we develop an interactive tool to elicit fairness preferences by projecting the complex decision space into a simple one-dimensional choice: setting risk thresholds. This design allows individuals to directly manipulate algorithm parameters and visualize the resulting tradeoffs between competing fairness metrics. Across two preregistered online experiments (n = 1, 563) in a healthcare context, we demonstrate that lay participants consistently identify efficient, near-optimal trade-offs that approach the Pareto frontier. Yet, their specific preferences are fragile: they shift significantly based on seemingly innocuous details of the information design, such as the order in which metrics are presented. These findings show that (1) our framework can effectively reveal fairness preferences and (2) the design of the elicitation process itself can fundamentally shape how fairness is perceived.

  • Open Access

    Tractable Agreement Protocols

    by Natalie Collina, Surbhi Goel, Varun Gupta, Aaron Roth


  • Open Access

    Resolving the Reference Class Problem at Scale

    by Aaron Roth, Alexander Williams Tolbert

    We draw a distinction between the traditional reference class problem, which describes an obstruction to estimating a single individual probability—which we rename the individual reference class problem —and what we call the reference class problem at scale , which can result when using tools from statistics and machine learning to systematically make predictions about many individual probabilities simultaneously. We argue that scale actually helps to mitigate the reference class problem, and purely statistical tools can be used to efficiently minimize the reference class problem at scale, even though they cannot be used to solve the individual reference class problem.


  • Open Access

    Repeated Contracting with Multiple Non-Myopic Agents: Policy Regret and Limited Liability

    by Natalie Collina, Varun Gupta, Aaron Roth

    We study a repeated contracting setting in which a Principal adaptively chooses amongst k Agents at each of T rounds. The Agents are non-myopic, and so a mechanism for the Principal induces a T-round extensive form game amongst the Agents. We give several results aimed at understanding an under-explored aspect of contract theory --- the game induced when choosing an Agent to contract with. First, we show that this game admits a pure-strategy non-responsive equilibrium amongst the Agents --- informally an equilibrium in which the Agent's actions depend on the history of realized states of nature, but not on the history of each other's actions, and so avoids the complexities of collusion and threats. Next, we show that if the Principal selects Agents using a monotone bandit algorithm, then for any concave contract, in any such equilibrium, the Principal obtains no regret to contracting with the best Agent in hindsight --- not just given their realized actions, but also to the counterfactual world in which they had offered a guaranteed T-round contract to the best Agent in hindsight, which would have induced a different sequence of actions. Finally, we show that if the Principal selects Agents using a monotone bandit algorithm which guarantees no swap-regret, then the Principal can additionally offer only limited liability contracts (in which the Agent never needs to pay the Principal) while getting no-regret to the counterfactual world in which she offered a linear contract to the best Agent in hindsight --- despite the fact that linear contracts are not limited liability. We instantiate this theorem by demonstrating the existence of a monotone no swap-regret bandit algorithm, which to our knowledge has not previously appeared in the literature.


  • Open Access

    Diversified Ensembling: An Experiment in Crowdsourced Machine Learning

    by Ira Globus-Harris, Declan Harrison, Michael Kearns, Pietro Perona, Aaron Roth

    Crowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In [12], the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task). Unlike traditional crowdsourcing, this allows for the diversification of participants’ efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern). We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams’ approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.


  • Open Access

    Multicalibrated Regression for Downstream Fairness

    by Ira Globus-Harris, Varun Gupta, Christopher Jung, Michael Kearns, Jamie Morgenstern, Aaron Roth

    We show how to take a regression function that is appropriately multicalibrated and efficiently post-process it into an approximately error minimizing classifier satisfying a large variety of fairness constraints. The post-processing requires no labeled data, and only a modest amount of unlabeled data and computation. The computational and sample complexity requirements of computing are comparable to the requirements for solving a single fair learning task optimally, but it can in fact be used to solve many different downstream fairness-constrained learning problems efficiently. Our post-processing method easily handles intersecting groups, generalizing prior work on post-processing regression functions to satisfy fairness constraints that only applied to disjoint groups. Our work extends recent work showing that multicalibrated regression functions are omnipredictors (i.e. can be post-processed to optimally solve unconstrained ERM problems) to constrained optimization problems.


  • Open Access

    Reconciling Individual Probability Forecasts✱

    by Aaron Roth, Alexander Tolbert, Scott Weinstein

    Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data—or on how to sample from a data distribution—cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of “reconciliation” that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions—providing an answer to what is sometimes called the predictive or model multiplicity problem.


  • Open Access

    Reconciling Individual Probability Forecasts

    by Aaron Roth, Alexander Tolbert, Scott Weinstein

    Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data -- or on how to sample from a data distribution -- cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of "reconciliation" that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions -- providing an answer to what is sometimes called the predictive or model multiplicity problem.