Nathan Kallus
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Nathan Kallus has a total of 119 co-authors. Here are the top co-authors by number of shared publications:
Andrew Bennett
Cornell Tech, Cornell University, New York, New York 10044
6 shared publications- Dawen Liang
Netflix, Inc., Los Gatos, CA, USA
6 shared publications - Dimitris Bertsimas
Sloan School of Management, Massachusetts Institute of Technology, Cambridge, USA
6 shared publications - Harald Steck
Netflix Inc., Los Gatos, CA, USA
6 shared publications - Masatoshi Uehara
Cornell University
6 shared publications - Xiaojie Mao
School of Economics and Management and Research Center for Contemporary Management, Tsinghua University , Beijing 100084
6 shared publications
Latest Publications
- 17 September 2026
- Published by Springer Nature in Nature Human Behaviour
- p. 1-12
The future of large-scale experiments and their challenges in the digital era
Large-scale randomized experiments involving millions of observations have transformed social science across academic and applied domains. These methods are now widely used in fields ranging from economics and political science to public health, education and digital product development. Drawing on the experience of industry practitioners and academic collaborators, this Perspective outlines six areas where the expanding scale and scope of experimental studies have introduced new challenges. We describe open problems related to organizational incentives and experimental governance; privacy, fairness and ethics; long-term impact estimation; time-adaptive experimental studies; heterogeneous treatment effects; and generative artificial intelligence. For each area, we summarize recent methodological advances, highlight ongoing limitations and suggest directions for future work. Our goal is to surface practical challenges that merit greater attention from the research community and to encourage sustained collaboration between academics and practitioners in shaping the future of experimental studies.
- 17 September 2026
- Published by Springer Nature in Nature Human Behaviour
- p. 1-12
The future of large-scale experiments and their challenges in the digital era
Large-scale randomized experiments involving millions of observations have transformed social science across academic and applied domains. These methods are now widely used in fields ranging from economics and political science to public health, education and digital product development. Drawing on the experience of industry practitioners and academic collaborators, this Perspective outlines six areas where the expanding scale and scope of experimental studies have introduced new challenges. We describe open problems related to organizational incentives and experimental governance; privacy, fairness and ethics; long-term impact estimation; time-adaptive experimental studies; heterogeneous treatment effects; and generative artificial intelligence. For each area, we summarize recent methodological advances, highlight ongoing limitations and suggest directions for future work. Our goal is to surface practical challenges that merit greater attention from the research community and to encourage sustained collaboration between academics and practitioners in shaping the future of experimental studies.
- Open Access
Does Weighting Improve Matrix Factorization for Recommender Systems?
Matrix factorization is a widely used approach for top-N recommendation and collaborative filtering. When implemented on implicit feedback data (such as clicks), a common heuristic is to upweight the observed interactions. This strategy has been shown to improve performance for certain algorithms. In this paper, we conduct a systematic study of various weighting schemes and matrix factorization algorithms. Somewhat surprisingly, we find that training with unweighted data can perform comparably to, and sometimes outperform, training with weighted data, especially for large models. This observation challenges the conventional wisdom. Nevertheless, we identify cases where weighting can be beneficial, particularly for models with lower capacity and specific regularization schemes. We also derive efficient algorithms for exactly minimizing several weighted objectives that were previously considered computationally intractable. Our work provides a comprehensive analysis of the interplay between weighting, regularization, and model capacity in matrix factorization for recommender systems.
Open AccessDoes Weighting Improve Matrix Factorization for Recommender Systems?
Matrix factorization is a widely used approach for top-N recommendation and collaborative filtering. When implemented on implicit feedback data (such as clicks), a common heuristic is to upweight the observed interactions. This strategy has been shown to improve performance for certain algorithms. In this paper, we conduct a systematic study of various weighting schemes and matrix factorization algorithms. Somewhat surprisingly, we find that training with unweighted data can perform comparably to, and sometimes outperform, training with weighted data, especially for large models. This observation challenges the conventional wisdom. Nevertheless, we identify cases where weighting can be beneficial, particularly for models with lower capacity and specific regularization schemes. We also derive efficient algorithms for exactly minimizing several weighted objectives that were previously considered computationally intractable. Our work provides a comprehensive analysis of the interplay between weighting, regularization, and model capacity in matrix factorization for recommender systems.
- Open Access
- 3 August 2025
- Published by Association for Computing Machinery (ACM)
- p. 4365-4374
Evaluating Decision Rules Across Many Weak Experiments
- 3 August 2025
- Published by Association for Computing Machinery (ACM)
- p. 4365-4374
Open AccessEvaluating Decision Rules Across Many Weak Experiments
- Open Access
- 20 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 165-176
CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policies
- 20 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 165-176
Open AccessCSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policies
- Open Access
Simulation-Based Inference for Adaptive Experiments
Multi-arm bandit experimental designs are increasingly being adopted over standard randomized trials due to their potential to improve outcomes for study participants, enable faster identification of the best-performing options, and/or enhance the precision of estimating key parameters. Current approaches for inference after adaptive sampling either rely on asymptotic normality under restricted experiment designs or underpowered martingale concentration inequalities that lead to weak power in practice. To bypass these limitations, we propose a simulation-based approach for conducting hypothesis tests and constructing confidence intervals for arm specific means and their differences. Our simulation-based approach uses positively biased nuisances to generate additional trajectories of the experiment, which we call \textit{simulation with optimism}. Using these simulations, we characterize the distribution potentially non-normal sample mean test statistic to conduct inference. We provide guarantees for (i) asymptotic type I error control, (ii) convergence of our confidence intervals, and (iii) asymptotic strong consistency of our estimator over a wide variety of common bandit designs. Our empirical results show that our approach achieves the desired coverage while reducing confidence interval widths by up to 50%, with drastic improvements for arms not targeted by the design.
Open AccessSimulation-Based Inference for Adaptive Experiments
Multi-arm bandit experimental designs are increasingly being adopted over standard randomized trials due to their potential to improve outcomes for study participants, enable faster identification of the best-performing options, and/or enhance the precision of estimating key parameters. Current approaches for inference after adaptive sampling either rely on asymptotic normality under restricted experiment designs or underpowered martingale concentration inequalities that lead to weak power in practice. To bypass these limitations, we propose a simulation-based approach for conducting hypothesis tests and constructing confidence intervals for arm specific means and their differences. Our simulation-based approach uses positively biased nuisances to generate additional trajectories of the experiment, which we call \textit{simulation with optimism}. Using these simulations, we characterize the distribution potentially non-normal sample mean test statistic to conduct inference. We provide guarantees for (i) asymptotic type I error control, (ii) convergence of our confidence intervals, and (iii) asymptotic strong consistency of our estimator over a wide variety of common bandit designs. Our empirical results show that our approach achieves the desired coverage while reducing confidence interval widths by up to 50%, with drastic improvements for arms not targeted by the design.
- Open Access
- 22 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 3323-3334
Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems
- 22 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 3323-3334
Open AccessCollaborative Retrieval for Large Language Model-based Conversational Recommender Systems
- Open Access
- 22 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 3885-3895
Does Weighting Improve Matrix Factorization for Recommender Systems?
- 22 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 3885-3895
Open AccessDoes Weighting Improve Matrix Factorization for Recommender Systems?
- Open Access
- 10 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 866-875
Reindex-Then-Adapt: Improving Large Language Models for Conversational Recommendation
Large Language Models (LLMs) are revolutionizing conversational recommender systems (CRS) by effectively indexing item content, understanding complex conversational contexts, and generating relevant item titles. However, the autoregressive nature of LLMs, which outputs item titles as a long sequence of subtokens, hinders the ability to efficiently obtain and control recommendations across the entire item set. This challenge in calculating probabilities over all items limits LLMs' potential, such as (1) limiting control over recommendation popularities and (2) preventing the synergy of marrying LLMs and traditional recommender systems (RecSys). To address this challenge, we propose the Reindex-Then-Adapt (RTA) framework. It consists of two steps: (1) Reindex: a lightweight network learns to condense multi-token item titles into single tokens within the LLM and distills LLM-generated recommendations as ranked lists. This bypasses the autoregressive nature of LLMs while trying to preserve their CRS abilities; (2) Adapt: LLMs after reindexing enable efficient adjustment of probability distributions over single-token titles, further enhanced through RecSys integration. RTA bridges the strengths of LLMs and RecSys, enabling understanding of complex queries as LLMs do, while efficiently controlling recommended item distributions as in traditional RecSys. We show the effectiveness of our RTA over base LLMs across three CRS datasets with negligible additional parameters.
- 10 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 866-875
Open AccessReindex-Then-Adapt: Improving Large Language Models for Conversational Recommendation
Large Language Models (LLMs) are revolutionizing conversational recommender systems (CRS) by effectively indexing item content, understanding complex conversational contexts, and generating relevant item titles. However, the autoregressive nature of LLMs, which outputs item titles as a long sequence of subtokens, hinders the ability to efficiently obtain and control recommendations across the entire item set. This challenge in calculating probabilities over all items limits LLMs' potential, such as (1) limiting control over recommendation popularities and (2) preventing the synergy of marrying LLMs and traditional recommender systems (RecSys). To address this challenge, we propose the Reindex-Then-Adapt (RTA) framework. It consists of two steps: (1) Reindex: a lightweight network learns to condense multi-token item titles into single tokens within the LLM and distills LLM-generated recommendations as ranked lists. This bypasses the autoregressive nature of LLMs while trying to preserve their CRS abilities; (2) Adapt: LLMs after reindexing enable efficient adjustment of probability distributions over single-token titles, further enhanced through RecSys integration. RTA bridges the strengths of LLMs and RecSys, enabling understanding of complex queries as LLMs do, while efficiently controlling recommended item distributions as in traditional RecSys. We show the effectiveness of our RTA over base LLMs across three CRS datasets with negligible additional parameters.
- 1 February 2025
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Mathematics of Operations Research
- Vol. 50  (1) , 633-655
Fast Rates for the Regret of Offline Reinforcement Learning
We study the regret of offline reinforcement learning in an infinite-horizon discounted Markov decision process (MDP). While existing analyses of common approaches, such as fitted Q-iteration (FQI), suggest root-n convergence for regret, empirical behavior exhibits much faster convergence. In this paper, we present a finer regret analysis that exactly characterizes this phenomenon by providing fast rates for the regret convergence. First, we show that given any estimate for the optimal quality function, the regret of the policy it defines converges at a rate given by the exponentiation of the estimate’s pointwise convergence rate, thus speeding up the rate. The level of exponentiation depends on the level of noise in the decision-making problem, rather than the estimation problem. We establish such noise levels for linear and tabular MDPs as examples. Second, we provide new analyses of FQI and Bellman residual minimization to establish the correct pointwise convergence guarantees. As specific cases, our results imply one-over-n rates in linear cases and exponential-in-n rates in tabular cases. We extend our findings to general function approximation by extending our results to regret guarantees based on Lp-convergence rates for estimating the optimal quality function rather than pointwise rates, where L2 guarantees for nonparametric estimation can be ensured under mild conditions. Funding: This work was supported by the Division of Information and Intelligent Systems, National Science Foundation [Grant 1846210].
- 1 February 2025
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Mathematics of Operations Research
- Vol. 50  (1) , 633-655
Fast Rates for the Regret of Offline Reinforcement Learning
We study the regret of offline reinforcement learning in an infinite-horizon discounted Markov decision process (MDP). While existing analyses of common approaches, such as fitted Q-iteration (FQI), suggest root-n convergence for regret, empirical behavior exhibits much faster convergence. In this paper, we present a finer regret analysis that exactly characterizes this phenomenon by providing fast rates for the regret convergence. First, we show that given any estimate for the optimal quality function, the regret of the policy it defines converges at a rate given by the exponentiation of the estimate’s pointwise convergence rate, thus speeding up the rate. The level of exponentiation depends on the level of noise in the decision-making problem, rather than the estimation problem. We establish such noise levels for linear and tabular MDPs as examples. Second, we provide new analyses of FQI and Bellman residual minimization to establish the correct pointwise convergence guarantees. As specific cases, our results imply one-over-n rates in linear cases and exponential-in-n rates in tabular cases. We extend our findings to general function approximation by extending our results to regret guarantees based on Lp-convergence rates for estimating the optimal quality function rather than pointwise rates, where L2 guarantees for nonparametric estimation can be ensured under mild conditions. Funding: This work was supported by the Division of Information and Intelligent Systems, National Science Foundation [Grant 1846210].
- 11 November 2024
- Published by Springer Nature in Machine Learning
- Vol. 113  (11-12) , 8347-8370
Adjusting regression models for conditional uncertainty calibration
- 11 November 2024
- Published by Springer Nature in Machine Learning
- Vol. 113  (11-12) , 8347-8370
Adjusting regression models for conditional uncertainty calibration