Ivan Sekulić
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Ivan Sekulić has a total of 23 co-authors. Here are the top co-authors by number of shared publications:
- Fabio Crestani
Università Della Svizzera italiana, Lugano, Switzerland
9 shared publications - Krisztian Balog
University of Stavanger, Stavanger, Norway
3 shared publications - Mohammad Aliannejadi
University of Amsterdam, Amstedam, Netherlands
3 shared publications Fabio Crestani
Università della Svizzera Italiana
2 shared publications- Jeffrey Dalton
The University of Edinburgh, Edinburgh, United Kingdom
2 shared publications Mohammad Aliannejadi
University of Amsterdam
2 shared publications
Latest Publications
- Open Access
- 7 February 2025
- Published by Qeios Ltd
Review of: "MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems"
The work presented in the article is timely and important, as the rise of RAG-based methods continues. The benchmark consists of 110 human-generated, multi-turn interactions, with each conversational turn involving a passage retrieval and an answer g...
- 7 February 2025
- Published by Qeios Ltd
Open AccessReview of: "MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems"
The work presented in the article is timely and important, as the rise of RAG-based methods continues. The benchmark consists of 110 human-generated, multi-turn interactions, with each conversational turn involving a passage retrieval and an answer g...
- Open Access
- 8 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 72-81
Simulating Conversational Search Users with Parameterized Behavior
User simulation is emerging as a promising direction towards scalable and reliable training and evaluation of conversational search systems. As such, the simulated user assumes the user's role in interaction with the system and aims to satisfy its information needs by querying, answering clarifying questions, and providing feedback. While recent research made significant progress in generating user's utterances, it remained limited to simulating the average user. In other words, state-of-the-art simulators do not take into account differences that exist between real users, such as personality and behavioral traits. To this end, we propose a framework for incorporating behavioral traits into a generative user simulator for conversational search. Specifically, we utilize in-context learning to embed behavioral traits, such as cooperativeness and politeness, into the LLM-based simulator. The framework, dubbed ParamConvSim, parametrizes certain behavioral traits and allows tuning the degree of a single trait present in the simulated user. In this paper, we design a user model, modeling the user's patience, cooperativeness, and politeness. In addition, we present and analyze the effectiveness of conversational search systems when interacting with different such simulated users. Results suggest that different simulated users indeed interact differently with the search system, leading to different system effectiveness levels.
- 8 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 72-81
Open AccessSimulating Conversational Search Users with Parameterized Behavior
User simulation is emerging as a promising direction towards scalable and reliable training and evaluation of conversational search systems. As such, the simulated user assumes the user's role in interaction with the system and aims to satisfy its information needs by querying, answering clarifying questions, and providing feedback. While recent research made significant progress in generating user's utterances, it remained limited to simulating the average user. In other words, state-of-the-art simulators do not take into account differences that exist between real users, such as personality and behavioral traits. To this end, we propose a framework for incorporating behavioral traits into a generative user simulator for conversational search. Specifically, we utilize in-context learning to embed behavioral traits, such as cooperativeness and politeness, into the LLM-based simulator. The framework, dubbed ParamConvSim, parametrizes certain behavioral traits and allows tuning the degree of a single trait present in the simulated user. In this paper, we design a user model, modeling the user's patience, cooperativeness, and politeness. In addition, we present and analyze the effectiveness of conversational search systems when interacting with different such simulated users. Results suggest that different simulated users indeed interact differently with the search system, leading to different system effectiveness levels.
- 6 December 2024
- Published by Springer Nature
- p. 246-260
Clarifying Questions Generation for Conversational Search Based on “People Also Ask” Feature
- 6 December 2024
- Published by Springer Nature
- p. 246-260
Clarifying Questions Generation for Conversational Search Based on “People Also Ask” Feature
- Open Access
- 17 May 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 15 (3) , 1-22
Analysing Utterances in LLM-Based User Simulation for Conversational Search
Clarifying underlying user information needs by asking clarifying questions is an important feature of modern conversational search systems. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In our recent work, we proposed an approach to tackle these issues with a user simulator,USi. Given a description of an information need,USiis capable of automatically answering clarifying questions about the topic throughout the search session. However, while the answers generated byUSiare both in line with the underlying information need and in natural language, a deeper understanding of such utterances is lacking. Thus, in this work, we explore utterance formulation of large language model (LLM)–based user simulators. To this end, we first analyze the differences betweenUSi, based on GPT-2, and the next generation of generative LLMs, such as GPT-3. Then, to gain a deeper understanding of LLM-based utterance generation, we compare the generated answers to the recently proposed set of patterns of human-based query reformulations. Finally, we discuss potential applications as well as limitations of LLM-based user simulators and outline promising directions for future work on the topic.
- 17 May 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 15 (3) , 1-22
Open AccessAnalysing Utterances in LLM-Based User Simulation for Conversational Search
Clarifying underlying user information needs by asking clarifying questions is an important feature of modern conversational search systems. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In our recent work, we proposed an approach to tackle these issues with a user simulator,USi. Given a description of an information need,USiis capable of automatically answering clarifying questions about the topic throughout the search session. However, while the answers generated byUSiare both in line with the underlying information need and in natural language, a deeper understanding of such utterances is lacking. Thus, in this work, we explore utterance formulation of large language model (LLM)–based user simulators. To this end, we first analyze the differences betweenUSi, based on GPT-2, and the next generation of generative LLMs, such as GPT-3. Then, to gain a deeper understanding of LLM-based utterance generation, we compare the generated answers to the recently proposed set of patterns of human-based query reformulations. Finally, we discuss potential applications as well as limitations of LLM-based user simulators and outline promising directions for future work on the topic.
- 23 March 2024
- Published by Springer Nature
- p. 384-392
Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search
- 23 March 2024
- Published by Springer Nature
- p. 384-392
Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search
- Open Access
- 10 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 209-218
Towards Self-Contained Answers: Entity-Based Answer Rewriting in Conversational Search
Conversational Information Seeking (CIS) is an emerging paradigm for knowledge acquisition and exploratory search. Traditional web search interfaces enable easy exploration of entities, but this is limited in conversational settings due to the limited-bandwidth interface. This paper explore ways to rewrite answers in CIS, so that users can understand them without having to resort to external services or sources. Specifically, we focus on salient entities—entities that are central to understanding the answer. As our first contribution, we create a dataset of conversations annotated with entities for saliency. Our analysis of the collected data reveals that the majority of answers contain salient entities. As our second contribution, we propose two answer rewriting strategies aimed at improving the overall user experience in CIS. One approach expands answers with inline definitions of salient entities, making the answer self-contained. The other approach complements answers with follow-up questions, offering users the possibility to learn more about specific entities. Results of a crowdsourcing-based study indicate that rewritten answers are clearly preferred over the original ones. We also find that inline definitions tend to be favored over follow-up questions, but this choice is highly subjective, thereby providing a promising future direction for personalization.
- 10 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 209-218
Open AccessTowards Self-Contained Answers: Entity-Based Answer Rewriting in Conversational Search
Conversational Information Seeking (CIS) is an emerging paradigm for knowledge acquisition and exploratory search. Traditional web search interfaces enable easy exploration of entities, but this is limited in conversational settings due to the limited-bandwidth interface. This paper explore ways to rewrite answers in CIS, so that users can understand them without having to resort to external services or sources. Specifically, we focus on salient entities—entities that are central to understanding the answer. As our first contribution, we create a dataset of conversations annotated with entities for saliency. Our analysis of the collected data reveals that the majority of answers contain salient entities. As our second contribution, we propose two answer rewriting strategies aimed at improving the overall user experience in CIS. One approach expands answers with inline definitions of salient entities, making the answer self-contained. The other approach complements answers with follow-up questions, offering users the possibility to learn more about specific entities. Results of a crowdsourcing-based study indicate that rewritten answers are clearly preferred over the original ones. We also find that inline definitions tend to be favored over follow-up questions, but this choice is highly subjective, thereby providing a promising future direction for personalization.
- Open Access
Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search
While the body of research directed towards constructing and generating clarifying questions in mixed-initiative conversational search systems is vast, research aimed at processing and comprehending users' answers to such questions is scarce. To this end, we present a simple yet effective method for processing answers to clarifying questions, moving away from previous work that simply appends answers to the original query and thus potentially degrades retrieval performance. Specifically, we propose a classifier for assessing usefulness of the prompted clarifying question and an answer given by the user. Useful questions or answers are further appended to the conversation history and passed to a transformer-based query rewriting module. Results demonstrate significant improvements over strong non-mixed-initiative baselines. Furthermore, the proposed approach mitigates the performance drops when non useful questions and answers are utilized.
Open AccessEstimating the Usefulness of Clarifying Questions and Answers for Conversational Search
While the body of research directed towards constructing and generating clarifying questions in mixed-initiative conversational search systems is vast, research aimed at processing and comprehending users' answers to such questions is scarce. To this end, we present a simple yet effective method for processing answers to clarifying questions, moving away from previous work that simply appends answers to the original query and thus potentially degrades retrieval performance. Specifically, we propose a classifier for assessing usefulness of the prompted clarifying question and an answer given by the user. Useful questions or answers are further appended to the conversation history and passed to a transformer-based query rewriting module. Results demonstrate significant improvements over strong non-mixed-initiative baselines. Furthermore, the proposed approach mitigates the performance drops when non useful questions and answers are utilized.
- Open Access
- 18 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 632-642
Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
- 18 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 632-642
Open AccessExploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
- Open Access
GRM: Generative Relevance Modeling Using Relevance-Aware Sample Estimation for Document Retrieval
Recent studies show that Generative Relevance Feedback (GRF), using text generated by Large Language Models (LLMs), can enhance the effectiveness of query expansion. However, LLMs can generate irrelevant information that harms retrieval effectiveness. To address this, we propose Generative Relevance Modeling (GRM) that uses Relevance-Aware Sample Estimation (RASE) for more accurate weighting of expansion terms. Specifically, we identify similar real documents for each generated document and use a neural re-ranker to estimate their relevance. Experiments on three standard document ranking benchmarks show that GRM improves MAP by 6-9% and R@1k by 2-4%, surpassing previous methods.
Open AccessGRM: Generative Relevance Modeling Using Relevance-Aware Sample Estimation for Document Retrieval
Recent studies show that Generative Relevance Feedback (GRF), using text generated by Large Language Models (LLMs), can enhance the effectiveness of query expansion. However, LLMs can generate irrelevant information that harms retrieval effectiveness. To address this, we propose Generative Relevance Modeling (GRM) that uses Relevance-Aware Sample Estimation (RASE) for more accurate weighting of expansion terms. Specifically, we identify similar real documents for each generated document and use a neural re-ranker to estimate their relevance. Experiments on three standard document ranking benchmarks show that GRM improves MAP by 6-9% and R@1k by 2-4%, surpassing previous methods.
- 3 October 2022
- Published by SPIE-Intl Soc Optical Eng
- Vol. 12227, 122270F-15
Boosting quantum optics experiments with Bayesian optimization (Conference Presentation)
Manual optimization of experimental parameters can quickly become too complex and time-consuming if more than a few correlated parameters need to be adjusted. We discuss automating this process using Bayesian optimization. This machine learning-based method is particularly suitable because it can handle noisy measurements, performs a global search and requires relatively few experimental runs. We discuss the efficient, scalable implementation of Bayesian optimization, present practical applications for tuning experimental parameters, and compare it with other local and global heuristic methods to show its application range.
- 3 October 2022
- Published by SPIE-Intl Soc Optical Eng
- Vol. 12227, 122270F-15
Boosting quantum optics experiments with Bayesian optimization (Conference Presentation)
Manual optimization of experimental parameters can quickly become too complex and time-consuming if more than a few correlated parameters need to be adjusted. We discuss automating this process using Bayesian optimization. This machine learning-based method is particularly suitable because it can handle noisy measurements, performs a global search and requires relatively few experimental runs. We discuss the efficient, scalable implementation of Bayesian optimization, present practical applications for tuning experimental parameters, and compare it with other local and global heuristic methods to show its application range.