What Language Model to Train if You Have One Million GPU Hours?

Authors:
  • TSTeven Le Scao
    Teven Le Scao
  • TWThomas Wang
    Thomas Wang
  • DHDaniel Hesslow
    Daniel Hesslow
  • SBStas Bekman
    Stas Bekman
  • MBM Saiful Bari
    M Saiful Bari
  • SBStella Biderman
    Stella Biderman
  • HEHady Elsahar
    Hady Elsahar
  • NMNiklas Muennighoff
    Niklas Muennighoff
  • JPJason Phang
    Jason Phang
  • OPOfir Press
    Ofir Press
  • CRColin Raffel
    Colin Raffel
  • VSVictor Sanh
    Victor Sanh

Abstract
Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, Iz Beltagy. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.