Scaling Deep Network Training

Authors:

Abstract
Training a modern deep network at scale is no longer a single decision but three coupled ones: which optimizer to trust with millions or billions of parameters, what numerical precision to compute in, and how to spend a fixed compute budget between model size and data. This manuscript develops each decision in turn and then reconnects them. It traces the optimizer lineage from momentum-corrected SGD through Adam, AdamW, and the layer-wise adaptive methods (LARS, LAMB) that make very large batches trainable, develops the mixed-precision training loop and the loss-scaling mechanism that keeps it numerically stable, derives the linear learning-rate scaling rule and its documented limits, and closes with the compute-optimal scaling laws that reframe model size itself as a quantity to be optimized rather than maximized.