Quantitative Biology > Neurons and Cognition
[Submitted on 8 Aug 2023 (this version), latest version 13 Aug 2024 (v3)]
Title:Comparing Color Similarity Structures between Humans and LLMs via Unsupervised Alignment
View PDFAbstract:Large language models (LLMs), such as the General Pre-trained Transformer (GPT), have shown remarkable performance in various cognitive tasks. However, it remains unclear whether these models have the ability to accurately infer human perceptual representations. Previous research has addressed this question by quantifying correlations between similarity response patterns from humans and LLMs. Although it has been shown that the correlation between humans and LLMs is reasonably high, simple correlation analysis is inadequate to reveal the degree of detailed structural correspondence between humans and LLMs. Here, we use an unsupervised alignment method based on Gromov-Wasserstein optimal transport to assess the equivalence of similarity structures between humans and LLMs in a more detailed manner. As a tractable study, we compare the color similarity structures of humans (color-neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and GPT-4) by examining the similarity structures of 93 colors. Our results show that the similarity structures of color-neurotypical humans can be remarkably well-aligned to that of GPT-4 and, to a lesser extent, to that of GPT-3.5. These results contribute to our understanding of the ability of LLMs to accurately infer human perception, and highlight the potential of unsupervised alignment methods to reveal detailed structural equivalence or differences that cannot be detected by simple correlation analysis.
Submission history
From: Genji Kawakita [view email][v1] Tue, 8 Aug 2023 16:32:41 UTC (6,472 KB)
[v2] Thu, 27 Jun 2024 10:41:08 UTC (2,735 KB)
[v3] Tue, 13 Aug 2024 17:11:25 UTC (2,735 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.