Can LLMs Simulate Target Users in Visualization Case Studies?

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Case studies are central to evaluating visualization research as they provide evidence for how the developed approaches support real users, real analytical work, and real data. Conducting such studies can be challenging, since target users with relevant domain expertise are often scarce or even entirely unavailable, while the visualization researchers may lack the required expertise to perform the evaluation themselves. Current research on Large Language Models (LLMs) has shown their strong reasoning ability and usefulness in domain‐specific downstream tasks, provoking the question: to what degree and with what capacity can LLMs fill the role of the target users in visualization case studies? We investigate this question by evaluating how LLMs can participate across different phases of visualization case studies. We propose a conceptual framework that explores the different levels of LLM involvement and potential roles when simulating target users. To sketch where the LLM substitution is most plausible, we embed our suggested integration of LLMs within the nested model for visualization theory and design. For evaluation, we replicate case studies from published and unpublished visualization research using multiple state‐of‐the‐art LLMs and compare the model‐generated insights to those reported in the papers. Our results show that LLMs can often generate plausible and sometimes even novel interpretations of visual patterns when used for result analysis and validation, but can also struggle when highly contextual, domain‐specific knowledge is required.
Funding Information
  • Deutsche Forschungsgemeinschaft (251654672—TRR 161)

This publication has 50 references indexed in Scilit: