Gromov–Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models
Open Access
- 10 July 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-10
Abstract
Large Language Models (LLMs), such as the General Pre-trained Transformer (GPT), have shown remarkable performance in various cognitive tasks. However, it remains unclear whether these models have the ability to accurately infer human perceptual representations. Previous research has addressed this question by quantifying correlations between similarity response patterns of humans and LLMs. Correlation provides a measure of similarity, but it relies pre-defined item labels and does not distinguish category- and item- level similarity, falling short of characterizing detailed structural correspondence between humans and LLMs. To assess their structural equivalence in more detail, we propose the use of an unsupervised alignment method based on Gromov–Wasserstein optimal transport (GWOT). GWOT allows for the comparison of similarity structures without relying on pre-defined label correspondences and can reveal fine-grained structural similarities and differences that may not be detected by simple correlation analysis. Using a large dataset of similarity judgments of 93 colors, we compared the color similarity structures of humans (color-neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and GPT-4). Our results show that the similarity structure of color-neurotypical participants can be remarkably well aligned with that of GPT-4 and, to a lesser extent, to that of GPT-3.5. These results contribute to the methodological advancements of comparing LLMs with human perception, and highlight the potential of unsupervised alignment methods to reveal detailed structural correspondences.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Unsupervised Alignment in Neuroscience: Introducing a Toolbox for Gromov-Wasserstein Optimal TransportbioRxiv, 2023
- Using cognitive psychology to understand GPT-3Proceedings of the National Academy of Sciences, 2023
- Revealing the multidimensional mental representations of natural objects underlying human similarity judgementsNature Human Behaviour, 2020
- Gromov-Wasserstein optimal transport to align single-cell multi-omics databioRxiv, 2020
- Acquisition of the Meaning of the Word Orange Requires Understanding of the Meanings of Red, Pink, and Purple: Constructing a Lexicon as a Connected SystemCognitive Science, 2020
- A method for identifying color vision deficiency malingeringGraefe's Archive for Clinical and Experimental Ophthalmology, 2016
- Representational geometry: integrating cognition, computation, and the brainTrends in Cognitive Sciences, 2013
- Gromov–Wasserstein Distances and the Metric Approach to Object MatchingFoundations of Computational Mathematics, 2011
- Russian blues reveal effects of language on color discriminationProceedings of the National Academy of Sciences, 2007
- The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observationsColor Research & Application, 2004