NKNathan Kallus

Nathan Kallus

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Nathan Kallus has a total of 119 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • The future of large-scale experiments and their challenges in the digital era

    by David Holtz, Iavor Bojinov, Ramesh Johari, Nathan Kallus, Apoorva Lal, Sathya Anand, Kyle Carlson, Brent Cohn, Tom Cunningham, Alex Deng, et al.

    Large-scale randomized experiments involving millions of observations have transformed social science across academic and applied domains. These methods are now widely used in fields ranging from economics and political science to public health, education and digital product development. Drawing on the experience of industry practitioners and academic collaborators, this Perspective outlines six areas where the expanding scale and scope of experimental studies have introduced new challenges. We describe open problems related to organizational incentives and experimental governance; privacy, fairness and ethics; long-term impact estimation; time-adaptive experimental studies; heterogeneous treatment effects; and generative artificial intelligence. For each area, we summarize recent methodological advances, highlight ongoing limitations and suggest directions for future work. Our goal is to surface practical challenges that merit greater attention from the research community and to encourage sustained collaboration between academics and practitioners in shaping the future of experimental studies.

  • Open Access

    Does Weighting Improve Matrix Factorization for Recommender Systems?

    by Alex Ayoub, Samuel Robertson, Dawen Liang, Harald Steck, Nathan Kallus

    Matrix factorization is a widely used approach for top-N recommendation and collaborative filtering. When implemented on implicit feedback data (such as clicks), a common heuristic is to upweight the observed interactions. This strategy has been shown to improve performance for certain algorithms. In this paper, we conduct a systematic study of various weighting schemes and matrix factorization algorithms. Somewhat surprisingly, we find that training with unweighted data can perform comparably to, and sometimes outperform, training with weighted data, especially for large models. This observation challenges the conventional wisdom. Nevertheless, we identify cases where weighting can be beneficial, particularly for models with lower capacity and specific regularization schemes. We also derive efficient algorithms for exactly minimizing several weighted objectives that were previously considered computationally intractable. Our work provides a comprehensive analysis of the interplay between weighting, regularization, and model capacity in matrix factorization for recommender systems.

  • Open Access

    Evaluating Decision Rules Across Many Weak Experiments

    by Winston Chou, Colin Gray, Nathan Kallus, Aurélien Bibaut, Simon Ejdemyr


  • Open Access

    CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policies

    by Brian Cho, Ana-Roxana Pop, Kyra Gan, Sam Corbett-Davies, Israel Nir, Ariel Evnine, Nathan Kallus


  • Open Access

    Simulation-Based Inference for Adaptive Experiments

    by Brian M Cho, Aurélien Bibaut, Nathan Kallus

    Multi-arm bandit experimental designs are increasingly being adopted over standard randomized trials due to their potential to improve outcomes for study participants, enable faster identification of the best-performing options, and/or enhance the precision of estimating key parameters. Current approaches for inference after adaptive sampling either rely on asymptotic normality under restricted experiment designs or underpowered martingale concentration inequalities that lead to weak power in practice. To bypass these limitations, we propose a simulation-based approach for conducting hypothesis tests and constructing confidence intervals for arm specific means and their differences. Our simulation-based approach uses positively biased nuisances to generate additional trajectories of the experiment, which we call \textit{simulation with optimism}. Using these simulations, we characterize the distribution potentially non-normal sample mean test statistic to conduct inference. We provide guarantees for (i) asymptotic type I error control, (ii) convergence of our confidence intervals, and (iii) asymptotic strong consistency of our estimator over a wide variety of common bandit designs. Our empirical results show that our approach achieves the desired coverage while reducing confidence interval widths by up to 50%, with drastic improvements for arms not targeted by the design.

  • Open Access

    Collaborative Retrieval for Large Language Model-based Conversational Recommender Systems

    by Yaochen Zhu, Chao Wan, Harald Steck, Dawen Liang, Yesu Feng, Nathan Kallus, Jundong Li


  • Open Access

    Does Weighting Improve Matrix Factorization for Recommender Systems?

    by Alex Ayoub, Samuel Robertson, Dawen Liang, Harald Steck, Nathan Kallus

  • Open Access

    Reindex-Then-Adapt: Improving Large Language Models for Conversational Recommendation

    by Zhankui He, Zhouhang Xie, Harald Steck, Dawen Liang, Rahul Jha, Nathan Kallus, Julian McAuley

    Large Language Models (LLMs) are revolutionizing conversational recommender systems (CRS) by effectively indexing item content, understanding complex conversational contexts, and generating relevant item titles. However, the autoregressive nature of LLMs, which outputs item titles as a long sequence of subtokens, hinders the ability to efficiently obtain and control recommendations across the entire item set. This challenge in calculating probabilities over all items limits LLMs' potential, such as (1) limiting control over recommendation popularities and (2) preventing the synergy of marrying LLMs and traditional recommender systems (RecSys). To address this challenge, we propose the Reindex-Then-Adapt (RTA) framework. It consists of two steps: (1) Reindex: a lightweight network learns to condense multi-token item titles into single tokens within the LLM and distills LLM-generated recommendations as ranked lists. This bypasses the autoregressive nature of LLMs while trying to preserve their CRS abilities; (2) Adapt: LLMs after reindexing enable efficient adjustment of probability distributions over single-token titles, further enhanced through RecSys integration. RTA bridges the strengths of LLMs and RecSys, enabling understanding of complex queries as LLMs do, while efficiently controlling recommended item distributions as in traditional RecSys. We show the effectiveness of our RTA over base LLMs across three CRS datasets with negligible additional parameters.


  • Fast Rates for the Regret of Offline Reinforcement Learning

    by Yichun Hu, Nathan Kallus, Masatoshi Uehara

    We study the regret of offline reinforcement learning in an infinite-horizon discounted Markov decision process (MDP). While existing analyses of common approaches, such as fitted Q-iteration (FQI), suggest root-n convergence for regret, empirical behavior exhibits much faster convergence. In this paper, we present a finer regret analysis that exactly characterizes this phenomenon by providing fast rates for the regret convergence. First, we show that given any estimate for the optimal quality function, the regret of the policy it defines converges at a rate given by the exponentiation of the estimate’s pointwise convergence rate, thus speeding up the rate. The level of exponentiation depends on the level of noise in the decision-making problem, rather than the estimation problem. We establish such noise levels for linear and tabular MDPs as examples. Second, we provide new analyses of FQI and Bellman residual minimization to establish the correct pointwise convergence guarantees. As specific cases, our results imply one-over-n rates in linear cases and exponential-in-n rates in tabular cases. We extend our findings to general function approximation by extending our results to regret guarantees based on Lp-convergence rates for estimating the optimal quality function rather than pointwise rates, where L2 guarantees for nonparametric estimation can be ensured under mild conditions. Funding: This work was supported by the Division of Information and Intelligent Systems, National Science Foundation [Grant 1846210].


  • Adjusting regression models for conditional uncertainty calibration

    by Ruijiang Gao, Mingzhang Yin, James Mcinerney, Nathan Kallus