Aaron Roth
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Aaron Roth has a total of 42 co-authors. Here are the top co-authors by number of shared publications:
- Michael Kearns
Computer and Information Sciences, University of Pennsylvania, USA and Amazon Web Services Artificial Intelligence (AWS AI), USA
4 shared publications - Ira Globus-Harris
Computer and Information Sciences, University of Pennsylvania, USA and Amazon Web Services Artificial Intelligence (AWS AI), USA
3 shared publications - Natalie Collina
University of Pennsylvania, Philadelphia, USA
3 shared publications - Varun Gupta
University of Pennsylvania, Philadelphia, USA
3 shared publications - Zhiwei Steven Wu
Carnegie Mellon University
3 shared publications - Alexander Williams Tolbert
Emory University
2 shared publications
Latest Publications
- Open Access
- 29 December 2025
- Published by Elsevier in SSRN Electronic Journal
Perceptions of Fairness in Algorithmic Decision-Making
In algorithmic decision-making, it is often mathematically impossible to satisfy competing definitions of fairness simultaneously. This impossibility makes assessing stakeholders' fairness preferences a challenging but crucial exercise. In this paper, we develop an interactive tool to elicit fairness preferences by projecting the complex decision space into a simple one-dimensional choice: setting risk thresholds. This design allows individuals to directly manipulate algorithm parameters and visualize the resulting tradeoffs between competing fairness metrics. Across two preregistered online experiments (n = 1, 563) in a healthcare context, we demonstrate that lay participants consistently identify efficient, near-optimal trade-offs that approach the Pareto frontier. Yet, their specific preferences are fragile: they shift significantly based on seemingly innocuous details of the information design, such as the order in which metrics are presented. These findings show that (1) our framework can effectively reveal fairness preferences and (2) the design of the elicitation process itself can fundamentally shape how fairness is perceived.
- 29 December 2025
- Published by Elsevier in SSRN Electronic Journal
Open AccessPerceptions of Fairness in Algorithmic Decision-Making
In algorithmic decision-making, it is often mathematically impossible to satisfy competing definitions of fairness simultaneously. This impossibility makes assessing stakeholders' fairness preferences a challenging but crucial exercise. In this paper, we develop an interactive tool to elicit fairness preferences by projecting the complex decision space into a simple one-dimensional choice: setting risk thresholds. This design allows individuals to directly manipulate algorithm parameters and visualize the resulting tradeoffs between competing fairness metrics. Across two preregistered online experiments (n = 1, 563) in a healthcare context, we demonstrate that lay participants consistently identify efficient, near-optimal trade-offs that approach the Pareto frontier. Yet, their specific preferences are fragile: they shift significantly based on seemingly innocuous details of the information design, such as the order in which metrics are presented. These findings show that (1) our framework can effectively reveal fairness preferences and (2) the design of the elicitation process itself can fundamentally shape how fairness is perceived.
- Open Access
- 15 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 1532-1543
Tractable Agreement Protocols
- 15 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 1532-1543
Open AccessTractable Agreement Protocols
- Open Access
- 14 April 2025
- Published by Cambridge University Press (CUP) in Philosophy of Science
- Vol. 92 (4) , 868-882
Resolving the Reference Class Problem at Scale
We draw a distinction between the traditional reference class problem, which describes an obstruction to estimating a single individual probability—which we rename the individual reference class problem —and what we call the reference class problem at scale , which can result when using tools from statistics and machine learning to systematically make predictions about many individual probabilities simultaneously. We argue that scale actually helps to mitigate the reference class problem, and purely statistical tools can be used to efficiently minimize the reference class problem at scale, even though they cannot be used to solve the individual reference class problem.
- 14 April 2025
- Published by Cambridge University Press (CUP) in Philosophy of Science
- Vol. 92 (4) , 868-882
Open AccessResolving the Reference Class Problem at Scale
We draw a distinction between the traditional reference class problem, which describes an obstruction to estimating a single individual probability—which we rename the individual reference class problem —and what we call the reference class problem at scale , which can result when using tools from statistics and machine learning to systematically make predictions about many individual probabilities simultaneously. We argue that scale actually helps to mitigate the reference class problem, and purely statistical tools can be used to efficiently minimize the reference class problem at scale, even though they cannot be used to solve the individual reference class problem.
- Open Access
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 466-488
Forecasting for Swap Regret for All Downstream Agents
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 466-488
Open AccessForecasting for Swap Regret for All Downstream Agents
- Open Access
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 640-668
Repeated Contracting with Multiple Non-Myopic Agents: Policy Regret and Limited Liability
We study a repeated contracting setting in which a Principal adaptively chooses amongst k Agents at each of T rounds. The Agents are non-myopic, and so a mechanism for the Principal induces a T-round extensive form game amongst the Agents. We give several results aimed at understanding an under-explored aspect of contract theory --- the game induced when choosing an Agent to contract with. First, we show that this game admits a pure-strategy non-responsive equilibrium amongst the Agents --- informally an equilibrium in which the Agent's actions depend on the history of realized states of nature, but not on the history of each other's actions, and so avoids the complexities of collusion and threats. Next, we show that if the Principal selects Agents using a monotone bandit algorithm, then for any concave contract, in any such equilibrium, the Principal obtains no regret to contracting with the best Agent in hindsight --- not just given their realized actions, but also to the counterfactual world in which they had offered a guaranteed T-round contract to the best Agent in hindsight, which would have induced a different sequence of actions. Finally, we show that if the Principal selects Agents using a monotone bandit algorithm which guarantees no swap-regret, then the Principal can additionally offer only limited liability contracts (in which the Agent never needs to pay the Principal) while getting no-regret to the counterfactual world in which she offered a linear contract to the best Agent in hindsight --- despite the fact that linear contracts are not limited liability. We instantiate this theorem by demonstrating the existence of a monotone no swap-regret bandit algorithm, which to our knowledge has not previously appeared in the literature.
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 640-668
Open AccessRepeated Contracting with Multiple Non-Myopic Agents: Policy Regret and Limited Liability
We study a repeated contracting setting in which a Principal adaptively chooses amongst k Agents at each of T rounds. The Agents are non-myopic, and so a mechanism for the Principal induces a T-round extensive form game amongst the Agents. We give several results aimed at understanding an under-explored aspect of contract theory --- the game induced when choosing an Agent to contract with. First, we show that this game admits a pure-strategy non-responsive equilibrium amongst the Agents --- informally an equilibrium in which the Agent's actions depend on the history of realized states of nature, but not on the history of each other's actions, and so avoids the complexities of collusion and threats. Next, we show that if the Principal selects Agents using a monotone bandit algorithm, then for any concave contract, in any such equilibrium, the Principal obtains no regret to contracting with the best Agent in hindsight --- not just given their realized actions, but also to the counterfactual world in which they had offered a guaranteed T-round contract to the best Agent in hindsight, which would have induced a different sequence of actions. Finally, we show that if the Principal selects Agents using a monotone bandit algorithm which guarantees no swap-regret, then the Principal can additionally offer only limited liability contracts (in which the Agent never needs to pay the Principal) while getting no-regret to the counterfactual world in which she offered a linear contract to the best Agent in hindsight --- despite the fact that linear contracts are not limited liability. We instantiate this theorem by demonstrating the existence of a monotone no swap-regret bandit algorithm, which to our knowledge has not previously appeared in the literature.
- Open Access
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 511-541
Efficient Prior-Free Mechanisms for No-Regret Agents
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 511-541
Open AccessEfficient Prior-Free Mechanisms for No-Regret Agents
- Open Access
- 3 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 529-545
Diversified Ensembling: An Experiment in Crowdsourced Machine Learning
Crowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In [12], the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task). Unlike traditional crowdsourcing, this allows for the diversification of participants’ efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern). We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams’ approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.
- 3 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 529-545
Open AccessDiversified Ensembling: An Experiment in Crowdsourced Machine Learning
Crowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In [12], the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task). Unlike traditional crowdsourcing, this allows for the diversification of participants’ efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern). We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams’ approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.
- Open Access
- 8 August 2023
- Published by Association for Computing Machinery (ACM)
- p. 259-286
Multicalibrated Regression for Downstream Fairness
We show how to take a regression function that is appropriately multicalibrated and efficiently post-process it into an approximately error minimizing classifier satisfying a large variety of fairness constraints. The post-processing requires no labeled data, and only a modest amount of unlabeled data and computation. The computational and sample complexity requirements of computing are comparable to the requirements for solving a single fair learning task optimally, but it can in fact be used to solve many different downstream fairness-constrained learning problems efficiently. Our post-processing method easily handles intersecting groups, generalizing prior work on post-processing regression functions to satisfy fairness constraints that only applied to disjoint groups. Our work extends recent work showing that multicalibrated regression functions are omnipredictors (i.e. can be post-processed to optimally solve unconstrained ERM problems) to constrained optimization problems.
- 8 August 2023
- Published by Association for Computing Machinery (ACM)
- p. 259-286
Open AccessMulticalibrated Regression for Downstream Fairness
We show how to take a regression function that is appropriately multicalibrated and efficiently post-process it into an approximately error minimizing classifier satisfying a large variety of fairness constraints. The post-processing requires no labeled data, and only a modest amount of unlabeled data and computation. The computational and sample complexity requirements of computing are comparable to the requirements for solving a single fair learning task optimally, but it can in fact be used to solve many different downstream fairness-constrained learning problems efficiently. Our post-processing method easily handles intersecting groups, generalizing prior work on post-processing regression functions to satisfy fairness constraints that only applied to disjoint groups. Our work extends recent work showing that multicalibrated regression functions are omnipredictors (i.e. can be post-processed to optimally solve unconstrained ERM problems) to constrained optimization problems.
- Open Access
- 12 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 101-110
Reconciling Individual Probability Forecasts✱
Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data—or on how to sample from a data distribution—cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of “reconciliation” that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions—providing an answer to what is sometimes called the predictive or model multiplicity problem.
- 12 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 101-110
Open AccessReconciling Individual Probability Forecasts✱
Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data—or on how to sample from a data distribution—cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of “reconciliation” that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions—providing an answer to what is sometimes called the predictive or model multiplicity problem.
- Open Access
Reconciling Individual Probability Forecasts
Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data -- or on how to sample from a data distribution -- cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of "reconciliation" that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions -- providing an answer to what is sometimes called the predictive or model multiplicity problem.
Open AccessReconciling Individual Probability Forecasts
Individual probabilities refer to the probabilities of outcomes that are realized only once: the probability that it will rain tomorrow, the probability that Alice will die within the next 12 months, the probability that Bob will be arrested for a violent crime in the next 18 months, etc. Individual probabilities are fundamentally unknowable. Nevertheless, we show that two parties who agree on the data -- or on how to sample from a data distribution -- cannot agree to disagree on how to model individual probabilities. This is because any two models of individual probabilities that substantially disagree can together be used to empirically falsify and improve at least one of the two models. This can be efficiently iterated in a process of "reconciliation" that results in models that both parties agree are superior to the models they started with, and which themselves (almost) agree on the forecasts of individual probabilities (almost) everywhere. We conclude that although individual probabilities are unknowable, they are contestable via a computationally and data efficient process that must lead to agreement. Thus we cannot find ourselves in a situation in which we have two equally accurate and unimprovable models that disagree substantially in their predictions -- providing an answer to what is sometimes called the predictive or model multiplicity problem.