Effective Theory of Transformers at Initialization
Preprint
Abstract
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- The Principles of Deep Learning Theory2022
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992
- Human behavior and the principle of least effort: An introduction to human ecology. By George Kingsley Zipf. Pp. xi, 573. Cambridge, Mass.: Addison-Wesley Press, 1949.Language, 1950