A Mathematical Explanation of Transformers

Authors:
Publisher website
Google Scholar
Abstract
The Transformer architecture has revolutionized the field of sequence modeling and underpins the recent breakthroughs in large language models (LLMs). However, a comprehensive mathematical theory that explains its structure and operations remains elusive. In this work, we propose a novel continuous framework that rigorously interprets the Transformer as a discretization of a structured integro-differential equation. Within this formulation, the self-attention mechanism emerges naturally as a nonlocal integral operator, and layer normalization is characterized as a projection to a time-dependent constraint. This operator-theoretic and variational perspective offers a unified and interpretable foundation for understanding the architecture’s core components, including attention, feedforward layers, and normalization. Our approach extends beyond previous theoretical analyses by embedding the entire Transformer operation in continuous domains for both token indices and feature dimensions. This leads to a principled and flexible framework that not only deepens on theoretical insight but also offers new directions for architecture design, analysis, and control-based interpretations. This new interpretation provides a step toward bridging the gap between deep learning architectures and continuous mathematical modeling, and contributes a foundational perspective to the ongoing development of interpretable and theoretically grounded neural network models.
Funding Information
  • NORCE
  • HKRGC (ECS 22302123)
  • HKRGC (GRF 12301925)
  • HKRGC (2025A0505000007)
  • Hetao Institute of Mathematics and Interdisciplinary Sciences (LU13300125)
  • HKRGC (LU13300125)
  • HKRGC (LU11309922)
  • ITF (MHP/054/22)
  • ITF (LUBGR105824)

This publication has 41 references indexed in Scilit: