OpenBMB: Big Model Systems for Large-Scale Representation Learning
Open Access
- 24 August 2023
- book chapter
- Published by Springer Nature
- p. 463-489
Abstract
Big pre-trained models (PTMs) have received increasing attention in recent years from academia and industry for their excellent performance on downstream tasks. However, huge computing power and sophisticated technical expertise are required to develop big models, discouraging many institutes and researchers. In order to facilitate the popularization of big models, we introduce OpenBMB, an open-source suite of big models, to break the barriers of computation and expertise of big model applications. In this chapter, we will introduce the core toolkits in OpenBMB, including BMTrain for efficient training, OpenPrompt and OpenDelta for efficient tuning, BMCook for efficient compression, and BMInf for efficient inference.Keywords
This publication has 19 references indexed in Scilit:
- Know what you don't need: Single-Shot Meta-Pruning for attention headsAI Open, 2021
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification2020
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Structured Pruning of Large Language Models2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Representation Learning for Natural Language Processing2020
- Patient Knowledge Distillation for BERT Model Compression2019
- Optimizing CUDA code by kernel fusion: application on BLASThe Journal of Supercomputing, 2015