Susan Zhang
Research Subjects
Research Interests
Susan Zhang has a total of 2 co-authors. Here are the top co-authors by number of shared publications:
- Emily Dinan
Meta AI Meta Platforms, Inc
1 shared publication - Sho Yaida
Meta AI Meta Platforms, Inc
1 shared publication
Latest Publications
- Open Access
Effective Theory of Transformers at Initialization
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.
Open AccessEffective Theory of Transformers at Initialization
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.