From Spikes to Attention: The Mathematical Odyssey of Neural Networks
Preprint
- 28 September 2026
- preprint
- Published by MDPI AG in Preprints.org
Abstract
This paper presents a comprehensive history of neural networks, tracing the journey from the earliest philosophical speculations about intelligence to modern large language models and diffusion models. It covers biological foundations, the McCulloch-Pitts neuron, Hebb’s rule, the Perceptron, the AI Winter, backpropagation, convolutional neural networks, recurrent networks, the Transformer revolution, and contemporary developments in generative AI, multimodal systems, and AI safety. This revised edition corrects factual errors, fills significant omissions, and expands the coverage of modern developments including scaling laws,RLHF, AlphaFold, vision transformers, and state-space models. Throughout, the mathematical theories underlying each development are explicitly presented, demonstrating that the history of neural networks is inseparable from the evolution of linear algebra, calculus, probability, and optimization.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Highly accurate protein structure prediction with AlphaFoldNature, 2021
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- The Elements of Statistical Learning2001
- Multilayer feedforward networks are universal approximatorsNeural Networks, 1989
- Learning representations by back-propagating errorsNature, 1986
- Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in positionBiological Cybernetics, 1980
- Impulses and Physiological States in Theoretical Models of Nerve MembraneBiophysical Journal, 1961
- The perceptron: A probabilistic model for information storage and organization in the brain.Psychological Review, 1958
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948
- A logical calculus of the ideas immanent in nervous activityBulletin of Mathematical Biology, 1943