Joseph M. Denning
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Joseph M. Denning has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Xiaohan (Hannah) Guo
Department of Psychology, University of Chicago, Chicago, IL, USA
2 shared publications Alex Martin
1 shared publicationAndrew Persichetti
1 shared publicationBryor Snefjella
Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
1 shared publication- Bryor Snefjella
McMaster University, Canada
1 shared publication Idan Asher Blank
1 shared publication
Latest Publications
- Open Access
Do Language Models Know Who Did What to Whom?
Language models (LMs) are commonly criticized for not “understanding” language. However, many critiques focus on cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring “who did what to whom” (thematic roles) in a sentence. Does the central training objective of LMs—word prediction—result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LMs that have been proposed as models of human language processing. The overall representational similarity of sentence pairs did not reflect whether they had the same agent/patient assignments or opposite agent/patient assignments. Furthermore, we found limited evidence that thematic role information was available in any subspace of hidden activations. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LMs can extract thematic roles but this information influences their representations weakly.
Open AccessDo Language Models Know Who Did What to Whom?
Language models (LMs) are commonly criticized for not “understanding” language. However, many critiques focus on cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring “who did what to whom” (thematic roles) in a sentence. Does the central training objective of LMs—word prediction—result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LMs that have been proposed as models of human language processing. The overall representational similarity of sentence pairs did not reflect whether they had the same agent/patient assignments or opposite agent/patient assignments. Furthermore, we found limited evidence that thematic role information was available in any subspace of hidden activations. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LMs can extract thematic roles but this information influences their representations weakly.
- Open Access
- 26 September 2024
- Published by Center for Open Science in PsyArXiv
Do Large Language Models know who did what to whom?
Large Language Models (LLMs) are commonly criticized for not “understanding” language. However, many critiques target cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring “who did what to whom” (thematic roles) in a sentence. Does the common training objective of LLMs—word prediction—result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LLMs. In contrast to human similarity judgments, in LLMs the overall representational similarity of sentence pairs reflected syntactic similarity but not whether their agent and patient assignments were identical vs. reversed. Furthermore, we found little evidence that thematic role information was available in any subset of hidden units. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LLMs can extract thematic roles but, unlike humans, this information influences their representations only weakly.
- 26 September 2024
- Published by Center for Open Science in PsyArXiv
Open AccessDo Large Language Models know who did what to whom?
Large Language Models (LLMs) are commonly criticized for not “understanding” language. However, many critiques target cognitive abilities that, in humans, are distinct from language processing. Here, we instead study a kind of understanding tightly linked to language: inferring “who did what to whom” (thematic roles) in a sentence. Does the common training objective of LLMs—word prediction—result in sentence representations that capture thematic roles? In two experiments, we characterized sentence representations in four LLMs. In contrast to human similarity judgments, in LLMs the overall representational similarity of sentence pairs reflected syntactic similarity but not whether their agent and patient assignments were identical vs. reversed. Furthermore, we found little evidence that thematic role information was available in any subset of hidden units. However, some attention heads robustly captured thematic roles, independently of syntax. Therefore, LLMs can extract thematic roles but, unlike humans, this information influences their representations only weakly.
- Open Access
- 21 November 2023
- Published by Center for Open Science in PsyArXiv
Taxonomic structure in a set of abstract concepts
A large portion of human knowledge comprises “abstract” concepts that lack readily perceivable properties (e.g., “love” and “justice”). Since abstract concepts lack such properties, they have historically been treated as an undifferentiated category of knowledge in the psychology and neuropsychology literatures. More recently, the categorical structure of abstract concepts is often explored using paradigms that ask participants to make explicit judgments about a set of concepts along dimensions that are predetermined by the experimenter. Such methods require the experimenter to select dimensions that are relevant to the concepts and further that people make explicit judgments that accurately reflect their mental representations. We bypassed these requirements by collecting two large sets of non-verbal and implicit judgments about which dimensions are relevant to the similarity between pairs of fifty abstract nouns to determine the representational space of the concepts. We then identified categories within the representational space using a clustering procedure that required categories to replicate across two independent data sets. In a separate experiment, we used automatic semantic priming to further validate the categories and to show that they are an improvement over categories that were defined within the same set of abstract concepts using explicit ratings along predetermined dimensions. These results demonstrate that abstract concepts can be characterized beyond their negative relation to concrete concepts and that categories of abstract concepts can be defined without using a priori dimensions for the concepts or explicit judgments from participants.
- 21 November 2023
- Published by Center for Open Science in PsyArXiv
Open AccessTaxonomic structure in a set of abstract concepts
A large portion of human knowledge comprises “abstract” concepts that lack readily perceivable properties (e.g., “love” and “justice”). Since abstract concepts lack such properties, they have historically been treated as an undifferentiated category of knowledge in the psychology and neuropsychology literatures. More recently, the categorical structure of abstract concepts is often explored using paradigms that ask participants to make explicit judgments about a set of concepts along dimensions that are predetermined by the experimenter. Such methods require the experimenter to select dimensions that are relevant to the concepts and further that people make explicit judgments that accurately reflect their mental representations. We bypassed these requirements by collecting two large sets of non-verbal and implicit judgments about which dimensions are relevant to the similarity between pairs of fifty abstract nouns to determine the representational space of the concepts. We then identified categories within the representational space using a clustering procedure that required categories to replicate across two independent data sets. In a separate experiment, we used automatic semantic priming to further validate the categories and to show that they are an improvement over categories that were defined within the same set of abstract concepts using explicit ratings along predetermined dimensions. These results demonstrate that abstract concepts can be characterized beyond their negative relation to concrete concepts and that categories of abstract concepts can be defined without using a priori dimensions for the concepts or explicit judgments from participants.