Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Authors:
- Stella BidermanSBStella Biderman
- Hailey SchoelkopfHSHailey Schoelkopf
- Quentin AnthonyQAQuentin Anthony
- Herbie BradleyHBHerbie Bradley
- Kyle O'BrienKOKyle O'Brien
- Eric HallahanEHEric Hallahan
- Mohammad Aflah KhanMKMohammad Aflah Khan
- Shivanshu PurohitSPShivanshu Purohit
- Usvsn Sai PrashanthUPUsvsn Sai Prashanth
- Edward RaffEREdward Raff
- Aviya SkowronASAviya Skowron
- Lintang SutawikaLSLintang Sutawika
Preprint
Abstract
How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at https://github.com/EleutherAI/pythia.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalizationNature Methods, 2024
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- Acquisition of chess knowledge in AlphaZeroProceedings of the National Academy of Sciences, 2022
- VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance2022
- The Values Encoded in Machine Learning Research2022
- A systematic evaluation of large language models of code2022
- Membership Inference Attacks on Machine Learning: A SurveyACM Computing Surveys, 2022
- Quality at a Glance: An Audit of Web-Crawled Multilingual DatasetsTransactions of the Association for Computational Linguistics, 2022
- Highly accurate protein structure prediction with AlphaFoldNature, 2021
- Unsupervised Cross-lingual Representation Learning at Scale2020