David Vos

Recent work on beyond-accuracy metrics for news recommendation has proposed RADio, a framework that operationalizes the concept of normative diversity from the social sciences into rank-aware evaluation metrics. RADio measures normative diversity as the divergence between ranked distributions over normative attributes of news articles. Vrijenhoek et al. caution that RADio’s conclusions may depend strongly on the dataset, cultural context, and the representation pipeline used to define such normative attributes. In this reproducibility study, we empirically study how robust RADio is to such contextual and representation choices. We evaluate the robustness of RADio along three axes. First, we extend RADio’s original evaluation from the English-language MIND dataset to the Danish EBNeRD and Norwegian Adressa news datasets, to test whether its metric behavior generalizes across editorial and cultural contexts. Second, we systematically vary key components of the attribute representation pipeline to examine how specific implementation choices influence RADio’s metric values. Finally, we analyze how RADio relates to recommendation accuracy by applying simple post-hoc re-ranking and quantifying the trade-offs between accuracy and normative diversity. Our results show that RADio outcomes are sensitive to both editorial context and representation design. Across datasets and representation choices, we observe that RADio can rank models’ normative diversity differently, and in some settings, the metric either saturates or collapses, limiting its ability to discriminate between models. We analyse why this happens, and give directions on designing the attribute representation pipeline. Furthermore, across all settings we consider, we can substantially increase RADio scores via re-ranking while incurring only modest losses in standard recommendation accuracy. These findings provide empirical support and nuance for RADio’s originally hypothesized contextual dependence, and underscore the need for context-aware evaluation frameworks when operationalizing normative diversity in news recommendations. In particular, the strong dependence on the attribute representation pipeline suggests that normative evaluation should be designed in close collaboration with editorial stakeholders. At the same time, our experiments indicate that, given appropriate representation choices, simple re-ranking can make normative-diversity-aware news recommendations practically viable. We share our code to facilitate reproducibility of our findings: https://github.com/davidvos/normative-diversity-repro /

Learning to rank (LTR) traditionally requires large-scale training data to generalize effectively. In low-data domains where expert annotation is scarce, the performance of LTR methods degrades sharply. Foundation models have alleviated similar data dependencies in other domains via in-context learning, but a foundation model for ranking with tabular features has not been explored yet. We propose prior-data fitted networks (PFNs) as a strong method for ranking in low-data settings. First, we demonstrate that PFNs, which are originally trained for classification, successfully outperform classification baselines on ranking data. Next, we evaluate PFNs as rankers, showing that they surpass state-of-the-art tuned baselines in low-data regimes. We introduce a novel sampling and inference scheme to obtain pairwise predictions from PFNs' native pointwise architecture, analogous to pairwise LTR. To address the limited context window of the transformers underlying PFNs, we propose a dynamic support set selection strategy for queries that scales PFNs beyond random subsampling. Our experimental results show that PFNs are an effective foundation model for ranking that provides significant gains when data is limited.

Recommender systems often struggle with long-tail distributions and limited item catalog exposure, where a small subset of popular items dominates recommendations. This challenge is especially critical in large-scale online retail settings with extensive and diverse product assortments. This paper introduces an approach to enhance catalog coverage without compromising recommendation quality in the existing digital recommendation pipeline at IKEA Retail. Drawing inspiration from recent advances in negative sampling to address popularity bias, we integrate contrastive learning with carefully selected negative samples. Through offline and online evaluations, we demonstrate that our method improves catalog coverage, ensuring a more diverse set of recommendations yet preserving strong recommendation performance.

Neural news recommender systems (RSs) have integrated language models (LMs) to encode news articles with rich textual information into representations, thereby improving the recommendation process. Most studies suggest that (i) news RSs achieve better performance with larger pre-trained language models (PLMs) than shallow language models (SLMs), and (ii) that large language models (LLMs) outperform PLMs. However, other studies indicate that PLMs sometimes lead to worse performance than SLMs. Thus, it remains unclear whether using larger LMs consistently improves the performance of news RSs. In this paper, we revisit, unify, and extend these comparisons of the effectiveness of LMs in news RSs using the real-world MIND dataset. We find that (i) larger LMs do not necessarily translate to better performance in news RSs, and (ii) they require stricter fine-tuning hyperparameter selection and greater computational resources to achieve optimal recommendation performance than smaller LMs. On the positive side, our experiments show that larger LMs lead to better recommendation performance for cold-start users: they alleviate dependency on extensive user interaction history and make recommendations more reliant on the news content.