A Study on Semantic Understanding of Large Language Models from the Perspective of Ambiguity Resolution
Open Access
- 7 July 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 165-170
Abstract
Generative large language models (LLMs) have demonstrated outstanding performance in language understanding and generation tasks. Whether current LLMs can truly understand meaning or not is an open question. Semantic disambiguation is an important indicator to test machine language ability. To this end, three experiments were conducted to test the ambiguity resolution ability. In Experiment 1, the LLMs were asked to make a semantic discrimination on ambiguity phrases. The ChatGPT (May 3 Version) and ChatGPT Plus (May 3 Version) showed a random level performance of discrimination. To make the task easier for the LLMs, ambiguity sentences were presented with context in the Experiment 2. While the LLMs unified the context to fill the position for ambiguity word, the performance of semantic discrimination on these sentences remained at random level. In addition to test the performance of LMMs, prediction is one of the key ability of LLMs. The prediction by unifying the context is one of the key skill for LLMs. Therefore, the Experiment 3 further test the similarity between the human prediction mechanism and the strong generation capabilities of LLMs. Weak similarities were only found under the condition of local constraint features gradually accumulating, specifically, the cosine similarity of predicted sentence semantic vectors and ending word semantic vectors negatively correlated with close probability only. These series of experiments demonstrate that, it is far away to claim that LLMs truly understand human sentences.Keywords
Funding Information
- National Social Science Fund of China (22&ZD298)
This publication has 5 references indexed in Scilit:
- Modelling the N400 brain potential as change in a probabilistic representation of meaningNature Human Behaviour, 2018
- The Predictive Mind2013
- The communicative function of ambiguity in languageCognition, 2012
- Aging in context: Age‐related changes in context use during language comprehensionPsychophysiology, 2005
- Brain potentials during reading reflect word expectancy and semantic associationNature, 1984