Hanne Vandenbroucke

News recommender systems (NRSs) are increasingly implemented in the news industry to enhance digital distribution, yet their development and evaluation are shaped by a complex interplay of organizational stakeholders with diverse objectives. This study investigates the real-world complexities of NRS implementation, considering the multitude of objectives on the NRS provider side by exploring how different stakeholder groups —from journalistic, commercial, and tech logic— conceptualize NRS goals and assess their performance. Through 22 semi-structured expert interviews across four news organizations in Europe, we uncover a ‘jungle of metrics’ where an overabundance of data insights does not necessarily lead to better decision-making. While traditional accuracy-based metrics like click-through rate (CTR) remain widely used, they fail to capture the broader strategic objectives that guide NRS adoption in industry. This paper identifies a critical gap between current academic evaluations of NRS and the business key performance indicators (KPIs) used in practice. Rather than evaluating NRS in isolation, stakeholders assess their contribution to overarching organizational goals, including reach, engagement, conversion, and retention. Our findings highlight the need for a more nuanced, multi-stakeholder approach to NRS evaluation that aligns with both business imperatives and journalistic values.

Recommender systems are increasingly used by news media organizations. Existing literature examines various aspects of news recommender systems (NRS) from a computational, user-centric, or normative perspective. Yet research advocates studying the complexities of real-world applications around NRS. Recently, a multi-stakeholder approach to NRS has been adopted, allowing to understand different stakeholder perspectives on NRS development and evaluation within the news organization. However, little research has been done on the different key performance indicators (KPIs) and metrics considered valuable by different stakeholders. Based on 11 interviews with professionals from two commercial news publishers, this paper demonstrates that stakeholders prioritize distinct KPIs and metrics related to the reach-engagement-conversion-retention funnel. The evaluation of NRS performance is often limited to short-term metrics like CTR, overlooking the multiplicity of stakeholders involved. Our findings reveal how different purposes, KPIs, and metrics are valued from the journalistic, commercial, and tech logic. In doing so, this paper contributes to the multi-stakeholder approach to NRS, advancing our understanding of the real-world complexity of NRS development and evaluation.