Abstract
We propose a new framework for measuring open-ended reasoning using large language models (LLMs), consisting of four stages: embedding and clustering, LLM assisted category definition, probabilistic scoring of category membership, and validation against human-coded benchmarks. We demonstrate the validity and usefulness of the framework through human survey responses and LLM-generated reasoning. We first validate the framework and show a naïve LLM fails to deliver stable and interpretable categories. We then demonstrate that continuous probability measures capture economically meaningful information that binary classification misses, including subjective uncertainty and cross-category structure. Finally, we apply the framework to LLM-generated reasoning to study variation across question formats, when reasoning data are no longer a binding constraint. In this application, the framework recovers both similarities and economically meaningful differences in reasoning across question formats.