Timo Breuer

Reproducibility lies at the basis of the empirical method: a novel approach will be widely adopted if its experimental results can be validated and reproduced by the community. Previous work on reproducibility in Information Retrieval (IR) has mainly addressed the reproducibility and replicability of offline experiments, with a few exceptions that replicate user studies. To the best of our knowledge, no previous work has investigated how reproducibility affects real users. In this paper, we do that by evaluating and comparing the reproducibility of an IR system both offline and online. We consider a reference system and generate a constellation of reproduced systems with varying parameters. We select 6 systems with different degrees of offline reproducibility. We then run a between-subjects online experiment with 280 participants and collect clicks to evaluate online reproducibility. Results show that real users do not perceive moderate variations of the reproducibility degree of systems, while they become relevant when the difference with the original system increases. Furthermore, we trained a click model to evaluate online reproducibility with simulated clicks. Results are not consistent with those from the user study, suggesting that better click models are needed to evaluate online reproducibility. Our data and source code are publicly available: https://github.com/angelogeninatti/reproducibilityLogs .

The rapid advancement of Large Language Models (LLMs) has introduced a paradigm shift in Information Retrieval (IR), moving beyond conventional keyword queries and ranked result lists. LLMs now play a critical role in the evolution of IR technologies and introduce new interaction forms like Retrieval-Augmented Generation, which is a more dynamic and interactive retrieval process that integrates various aspects of Information Access, like Question Answering, into the dialog between a searcher and the search engine. We explore the multi-faceted impact of LLMs on IR, particularly in three distinct layers where they have become an integral part of the retrieval process, namely the retrieval system and processing pipeline that can make use of a richer semantic representation using advanced language models, the interaction layer, and the broader IR ecosystem. For the latter, we focus on evaluation issues as well as bias, fairness, and ethical concerns. We also highlight some recent cases of using LLMs in the medical domain to demonstrate the impact on one specific domain.