LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models
Authors:
Open Access
- 10 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (1) , 503-513
Abstract
Evaluating large language models (LLMs) presents unique challenges. While automatic side-by-side evaluation, also known as LLM-as-a-judge, has become a promising solution, model developers and researchers face difficulties with scalability and interpretability when analyzing these evaluation outcomes. To address these challenges, we introduce LLM Comparator, a new visual analytics tool designed for side-by-side evaluations of LLMs. This tool provides analytical workflows that help users understand when and why one LLM outperforms or underperforms another, and how their responses differ. Through close collaboration with practitioners developing LLMs at Google, we have iteratively designed, developed, and refined the tool. Qualitative feedback from these users highlights that the tool facilitates in-depth analysis of individual examples while enabling users to visually overview and flexibly slice data. This empowers users to identify undesirable patterns, formulate hypotheses about model behavior, and gain insights for model improvement. LLM Comparator has been integrated into Google's LLM evaluation platforms and open-sourced.Keywords
Funding Information
- People + AI Research (PAIR) team at Google, and the users of LLM Comparator
This publication has 38 references indexed in Scilit:
- embComp: Visual Interactive Comparison of Vector EmbeddingsIEEE Transactions on Visualization and Computer Graphics, 2020
- A survey of visual analytics techniques for machine learningComputational Visual Media, 2020
- DECE: Decision Explorer with Counterfactual Explanations for Machine Learning ModelsIEEE Transactions on Visualization and Computer Graphics, 2020
- Machine Learning Testing: Survey, Landscapes and HorizonsIEEE Transactions on Software Engineering, 2020
- The What-If Tool: Interactive Probing of Machine Learning ModelsIEEE Transactions on Visualization and Computer Graphics, 2019
- RuleMatrix: Visualizing and Understanding Classifiers with RulesIEEE Transactions on Visualization and Computer Graphics, 2018
- Visual Analytics in Deep Learning: An Interrogative Survey for the Next FrontiersIEEE Transactions on Visualization and Computer Graphics, 2018
- ActiVis: Visual Exploration of Industry-Scale Deep Neural Network ModelsIEEE Transactions on Visualization and Computer Graphics, 2017
- The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations2003
- FCM: The fuzzy c-means clustering algorithmComputers & Geosciences, 1984