An enriched category theory of language: from syntax to semantics
Preprint
Abstract
State of the art language models return a natural language text continuation from any piece of input text. This ability to generate coherent text extensions implies significant sophistication, including a knowledge of grammar and semantics. In this paper, we propose a mathematical framework for passing from probability distributions on extensions of given texts, such as the ones learned by today's large language models, to an enriched category containing semantic information. Roughly speaking, we model probability distributions on texts as a category enriched over the unit interval. Objects of this category are expressions in language, and hom objects are conditional probabilities that one expression is an extension of another. This category is syntactical -- it describes what goes with what. Then, via the Yoneda embedding, we pass to the enriched category of unit interval-valued copresheaves on this syntactical category. This category of enriched copresheaves is semantic -- it is where we find meaning, logical operations such as entailment, and the building blocks for more elaborate semantic concepts.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Language Modeling with Reduced DensitiesCompositionality, 2021
- Tropical Geometry and Machine LearningProceedings of the IEEE, 2021
- Semantic Unification2014
- Tight spans, Isbell completions and semi-tropical modulesTheory and Applications of Categories, 2013
- From Frequency to Meaning: Vector Space Models of SemanticsJournal of Artificial Intelligence Research, 2010
- A Use Theory of MeaningPhilosophy and Phenomenological Research, 2004
- Conceptual Mathematics: A First Introduction to Categories.The American Mathematical Monthly, 1997
- Metric spaces, generalized logic, and closed categoriesRendiconti del Seminario Matematico e Fisico di Milano, 1973
- Adjointness in FoundationsDialectica, 1969