Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems
Preprint
Abstract
Evaluating retrieval-ranking systems is crucial for developing high-performing models. While online A/B testing is the gold standard, its high cost and risks to user experience require effective offline methods. However, relying on historical interaction data introduces biases-such as selection, exposure, conformity, and position biases-that distort evaluation metrics, driven by the Missing-Not-At-Random (MNAR) nature of user interactions and favoring popular or frequently exposed items over true user preferences. We propose a novel framework for robust offline evaluation of retrieval-ranking systems, transforming MNAR data into Missing-At-Random (MAR) through reweighting combined with black-box optimization, guided by neural estimation of information-theoretic metrics. Our contributions include (1) a causal formulation for addressing offline evaluation biases, (2) a system-agnostic debiasing framework, and (3) empirical validation of its effectiveness. This framework enables more accurate, fair, and generalizable evaluations, enhancing model assessment before deployment.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Invariant debiasing learning for recommendation via biased imputationInformation Processing & Management, 2024
- Bias and Debias in Recommender System: A Survey and Future DirectionsACM Transactions on Information Systems, 2023
- Counterfactual Phenotyping with Censored Time-to-Events2022
- Dual Unbiased Recommender Learning for Implicit Feedback2021
- Unifying Online and Counterfactual Learning to Rank: A Novel Counterfactual Estimator that Effectively Utilizes Online Interventions2021
- An Adversarial Approach to Improve Long-Tail Performance in Neural Collaborative Filtering2018
- Collaborative Metric Learning2017
- Top-N Recommendation with Missing Implicit Feedback2015
- Training and testing of recommender systems on data missing not at random2010
- Collaborative prediction and ranking with non-random missing data2009