Moving Beyond Next-Token Prediction: Transformers are Context-Sensitive Language Generators
Author:
Preprint
Abstract
Large Language Models (LLMs), powered by Transformers, have demonstrated human-like intelligence capabilities, yet their underlying mechanisms remain poorly understood. This paper presents a novel framework for interpreting LLMs as probabilistic left context-sensitive languages (CSLs) generators. We hypothesize that Transformers can be effectively decomposed into three fundamental components: context windows, attention mechanisms, and autoregressive generation frameworks. This decomposition allows for the development of more flexible and interpretable computational models, moving beyond the traditional view of attention and autoregression as inseparable processes. We argue that next-token predictions can be understood as probabilistic, dynamic approximations of left CSL production rules, providing an intuitive explanation for how simple token predictions can yield human-like intelligence outputs. Given that all CSLs are left context-sensitive (Penttonen, 1974), we conclude that Transformers stochastically approximate CSLs, which are widely recognized as models of human-like intelligence. This interpretation bridges the gap between Formal Language Theory and the observed generative power of Transformers, laying a foundation for future advancements in generative AI theory and applications. Our novel perspective on Transformer architectures will foster a deeper understanding of LLMs and their future potentials.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Unlocking the black box: an in-depth review on interpretability, explainability, and reliability in deep learningNeural Computing and Applications, 2024
- Statistical properties of probabilistic context-sensitive grammarsPhysical Review Research, 2024
- What Formal Languages Can Transformers Express? A SurveyTransactions of the Association for Computational Linguistics, 2024
- Explainability for Large Language Models: A SurveyACM Transactions on Intelligent Systems and Technology, 2024
- Representational Strengths and Limitations of Transformers2023
- Extracting automata from neural networks using active learningPeerJ Computer Science, 2021
- On the Computational Power of Transformers and Its Implications in Sequence Modeling2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Context-Sensitive Grammars and Linear-Bounded AutomataInternational Journal of Computer Network and Information Security, 2016
- Introduction to automata theory, languages, and computation, 2nd editionACM SIGACT News, 2001