Effect of LLM's Personality Traits on Query Generation
Open Access
- 8 December 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 249-258
Abstract
Large language models (LLMs) have demonstrated strong performance across various natural language processing tasks and are increasingly integrated into daily life. Just as personality traits are crucial in human communication, they could also play a significant role in the behavior of LLMs, for instance, in the context of Retrieval Augmented Generation. Previous studies have shown that Big Five personality traits could be applied to LLMs, but their specific effects on information retrieval tasks have not been sufficiently explored. This study aims to examine how personality traits assigned to LLM agents affect their query formulation behavior and search performance. We propose a method to accurately assign personality traits to LLM agents based on the Big Five theory and verify its accuracy using the IPIP-NEO-120 scale. We then design a query generation experiment using the NTCIR Ad-Hoc test collections and evaluate the search performance of queries generated by different LLM agents. The results show that our method successfully assigns all five personality traits to LLM agents as intended. Additionally, the query generation experiment suggests that the assigned traits did influence the length and vocabulary choices of generated queries. Finally, the retrieval effectiveness of the traits varied across test collections, showing a relative improvement ranging from -7.7% to +4.6%, but these differences were not statistically significant.Keywords
This publication has 18 references indexed in Scilit:
- Large Language Models can Accurately Predict Searcher Preferences2024
- Enhancing Human Annotation: Leveraging Large Language Models and Efficient Batch Processing2024
- Perspectives on Large Language Models for Relevance Judgment2023
- Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study2023
- Big five personality influences trajectories of information seeking behaviorPersonality and Individual Differences, 2021
- Declarative Experimentation in Information Retrieval using PyTerrier2020
- The impact of personality traits on users’ information-seeking behaviorInformation Processing & Management, 2017
- personality and information behavior in web searchProceedings of the Association for Information Science and Technology, 2016
- Measuring thirty facets of the Five Factor Model with a 120-item public domain inventory: Development of the IPIP-NEO-120Journal of Research in Personality, 2014
- Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval1994