Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Authors:
  • QAQuentin Anthony
    Quentin Anthony
  • HBHerbie Bradley
    Herbie Bradley
  • KOKyle O'Brien
    Kyle O'Brien
  • EHEric Hallahan
    Eric Hallahan
  • MKMohammad Aflah Khan
    Mohammad Aflah Khan
  • SPShivanshu Purohit
    Shivanshu Purohit
  • UPUsvsn Sai Prashanth
    Usvsn Sai Prashanth
  • EREdward Raff
    Edward Raff
  • ASAviya Skowron
    Aviya Skowron
  • LSLintang Sutawika
    Lintang Sutawika

Abstract
How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at https://github.com/EleutherAI/pythia.

This publication has 20 references indexed in Scilit: