Norbert Fuhr

We present a reusable dataset and accompanying infrastructure for studying human search behavior in Interactive Information Retrieval (IIR). The dataset combines detailed interaction logs from 61 participants (122 sessions) with user characteristics, including perceptual speed, topic-specific interest, search expertise, and demographic information. To facilitate reproducibility and reuse, we provide a fully documented study setup, a web-based perceptual speed test, and a framework for conducting similar user studies. Our work allows researchers to investigate individual and contextual factors affecting search behavior, and to develop or validate user simulators that account for such variability. We illustrate the datasets potential through an illustrative analysis and release all resources as open-access, supporting reproducible research and resource sharing in the IIR community.

The rapid advancement of Large Language Models (LLMs) has introduced a paradigm shift in Information Retrieval (IR), moving beyond conventional keyword queries and ranked result lists. LLMs now play a critical role in the evolution of IR technologies and introduce new interaction forms like Retrieval-Augmented Generation, which is a more dynamic and interactive retrieval process that integrates various aspects of Information Access, like Question Answering, into the dialog between a searcher and the search engine. We explore the multi-faceted impact of LLMs on IR, particularly in three distinct layers where they have become an integral part of the retrieval process, namely the retrieval system and processing pipeline that can make use of a richer semantic representation using advanced language models, the interaction layer, and the broader IR ecosystem. For the latter, we focus on evaluation issues as well as bias, fairness, and ethical concerns. We also highlight some recent cases of using LLMs in the medical domain to demonstrate the impact on one specific domain.

In this paper, we present a new approach to improving the relevance and reliability of medical information retrieval, which builds upon the concept of Level of Evidence (LoE). The LoE framework categorizes medical publications into seven distinct levels based on the underlying empirical evidence. Despite LoE framework's relevance in medical research and evidence-based practice, only few medical publications explicitly state their LoE. Therefore, we develop a classification model for automatically assigning LoE to medical publications, which successfully classifies over 26 million documents in MEDLINE database into LoE classes. The subsequent retrieval experiments on the TREC Precision Medicine datasets show substantial improvements in retrieval relevance, when LoE is used as a search filter.

This study investigates the potential of biomedical concepts—genes, diseases, and chemicals—in enhancing biomedical literature retrieval systems amidst the burgeoning volume of biomedical literature. We compare five traditional sparse approaches against hybrid BERT-based methods, to assess their effectiveness in leveraging biomedical concepts for improved retrieval accuracy. Our research poses critical questions on the necessity and efficiency of biomedical concepts in the era of advanced language models and evaluates if these concepts can further refine retrieval outcomes. Using datasets from Text REtrieval Conference Precision Medicine tracks (2017–2019), which are based on the Medline collection (30+ million biomedical publications) and NDCG@10 (Normalized Discounted Cumulative Gain) for evaluation, we demonstrate that biomedical concepts are indeed helpful for both hybrid and sparse retrieval strategies in the biomedical domain and that the performance of sparse and hybrid methods is comparable. We further demonstrate how our findings can be integrated into a live search system to support clinical practice.

This paper is a report of the Workshop on Simulations for Information Access (Sim4IA) workshop at SIGIR 2024. The workshop had two keynotes, a panel discussion, nine lightning talks, and two breakout sessions. Key takeaways were user simulation's importance in academia and industry, the possible bridging of online and offline evaluation, and the issues of organizing a companion shared task around user simulations for information access. We report on how we organized the workshop, provide a brief overview of what happened at the workshop, and summarize the main topics and findings of the workshop and future work.

Simulations in various forms have been used to evaluate information access systems, like search engines, recommender systems, or conversational agents. In the form of the Cranfield paradigm, a simulation setup is well-known in the IR community, but user simulations have recently gained interest. While user simulations help to reduce the complexity of evaluation experiments and help with reproducibility, they can also contribute to a better understanding of users. Building on recent developments in methods and toolkits, the Sim4IA workshop aims to bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.