ISIvan Sekulić

Ivan Sekulić

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Ivan Sekulić has a total of 23 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Review of: "MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems"

    by Ivan Sekulić

    The work presented in the article is timely and important, as the rise of RAG-based methods continues. The benchmark consists of 110 human-generated, multi-turn interactions, with each conversational turn involving a passage retrieval and an answer g...

  • Open Access

    Simulating Conversational Search Users with Parameterized Behavior

    by Ivan Sekulić, Lili Lu, Navdeep Singh Bedi, Fabio Crestani

    User simulation is emerging as a promising direction towards scalable and reliable training and evaluation of conversational search systems. As such, the simulated user assumes the user's role in interaction with the system and aims to satisfy its information needs by querying, answering clarifying questions, and providing feedback. While recent research made significant progress in generating user's utterances, it remained limited to simulating the average user. In other words, state-of-the-art simulators do not take into account differences that exist between real users, such as personality and behavioral traits. To this end, we propose a framework for incorporating behavioral traits into a generative user simulator for conversational search. Specifically, we utilize in-context learning to embed behavioral traits, such as cooperativeness and politeness, into the LLM-based simulator. The framework, dubbed ParamConvSim, parametrizes certain behavioral traits and allows tuning the degree of a single trait present in the simulated user. In this paper, we design a user model, modeling the user's patience, cooperativeness, and politeness. In addition, we present and analyze the effectiveness of conversational search systems when interacting with different such simulated users. Results suggest that different simulated users indeed interact differently with the search system, leading to different system effectiveness levels.


  • Open Access

    Analysing Utterances in LLM-Based User Simulation for Conversational Search

    by Ivan Sekulić, Mohammad Alinannejadi, Fabio Crestani

    Clarifying underlying user information needs by asking clarifying questions is an important feature of modern conversational search systems. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In our recent work, we proposed an approach to tackle these issues with a user simulator,USi. Given a description of an information need,USiis capable of automatically answering clarifying questions about the topic throughout the search session. However, while the answers generated byUSiare both in line with the underlying information need and in natural language, a deeper understanding of such utterances is lacking. Thus, in this work, we explore utterance formulation of large language model (LLM)–based user simulators. To this end, we first analyze the differences betweenUSi, based on GPT-2, and the next generation of generative LLMs, such as GPT-3. Then, to gain a deeper understanding of LLM-based utterance generation, we compare the generated answers to the recently proposed set of patterns of human-based query reformulations. Finally, we discuss potential applications as well as limitations of LLM-based user simulators and outline promising directions for future work on the topic.


  • Open Access

    Towards Self-Contained Answers: Entity-Based Answer Rewriting in Conversational Search

    by Ivan Sekulic, Krisztian Balog, Fabio Crestani

    Conversational Information Seeking (CIS) is an emerging paradigm for knowledge acquisition and exploratory search. Traditional web search interfaces enable easy exploration of entities, but this is limited in conversational settings due to the limited-bandwidth interface. This paper explore ways to rewrite answers in CIS, so that users can understand them without having to resort to external services or sources. Specifically, we focus on salient entities—entities that are central to understanding the answer. As our first contribution, we create a dataset of conversations annotated with entities for saliency. Our analysis of the collected data reveals that the majority of answers contain salient entities. As our second contribution, we propose two answer rewriting strategies aimed at improving the overall user experience in CIS. One approach expands answers with inline definitions of salient entities, making the answer self-contained. The other approach complements answers with follow-up questions, offering users the possibility to learn more about specific entities. Results of a crowdsourcing-based study indicate that rewritten answers are clearly preferred over the original ones. We also find that inline definitions tend to be favored over follow-up questions, but this choice is highly subjective, thereby providing a promising future direction for personalization.


  • Open Access

    Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search

    by Ivan Sekulić, Weronika Łajewska, Krisztian Balog, Fabio Crestani

    While the body of research directed towards constructing and generating clarifying questions in mixed-initiative conversational search systems is vast, research aimed at processing and comprehending users' answers to such questions is scarce. To this end, we present a simple yet effective method for processing answers to clarifying questions, moving away from previous work that simply appends answers to the original query and thus potentially degrades retrieval performance. Specifically, we propose a classifier for assessing usefulness of the prompted clarifying question and an answer given by the user. Useful questions or answers are further appended to the conversation history and passed to a transformer-based query rewriting module. Results demonstrate significant improvements over strong non-mixed-initiative baselines. Furthermore, the proposed approach mitigates the performance drops when non useful questions and answers are utilized.

  • Open Access

    Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond

    by Paul Owoicho, Ivan Sekulic, Mohammad Aliannejadi, Jeffrey Dalton, Fabio Crestani


  • Open Access

    GRM: Generative Relevance Modeling Using Relevance-Aware Sample Estimation for Document Retrieval

    by Iain Mackie, Ivan Sekulic, Shubham Chatterjee, Jeffrey Dalton, Fabio Crestani

    Recent studies show that Generative Relevance Feedback (GRF), using text generated by Large Language Models (LLMs), can enhance the effectiveness of query expansion. However, LLMs can generate irrelevant information that harms retrieval effectiveness. To address this, we propose Generative Relevance Modeling (GRM) that uses Relevance-Aware Sample Estimation (RASE) for more accurate weighting of expansion terms. Specifically, we identify similar real documents for each generated document and use a neural re-ranker to estimate their relevance. Experiments on three standard document ranking benchmarks show that GRM improves MAP by 6-9% and R@1k by 2-4%, surpassing previous methods.


  • Boosting quantum optics experiments with Bayesian optimization (Conference Presentation)

    by Philipp-Immanuel Schneider, Lin Zschiedrich, Martin Hammerschmidt, Lilli Kuen, Ivan Sekulic, Julien Kluge, Bastian Leykauf, Markus Krutzik, Sven Burger

    Manual optimization of experimental parameters can quickly become too complex and time-consuming if more than a few correlated parameters need to be adjusted. We discuss automating this process using Bayesian optimization. This machine learning-based method is particularly suitable because it can handle noisy measurements, performs a global search and requires relatively few experimental runs. We discuss the efficient, scalable implementation of Bayesian optimization, present practical applications for tuning experimental parameters, and compare it with other local and global heuristic methods to show its application range.