PUMA: Secure Inference of LLaMA-7B in Five Minutes
Preprint
Abstract
With ChatGPT as a representative, tons of companies have began to provide services based on large Transformers models. However, using such a service inevitably leak users' prompts to the model provider. Previous studies have studied secure inference for Transformer models using secure multiparty computation (MPC), where model parameters and clients' prompts are kept secret. Despite this, these frameworks are still limited in terms of model performance, efficiency, and deployment. To address these limitations, we propose framework PUMA to enable fast and secure Transformer model inference. Our framework designs high quality approximations for expensive functions such as GeLU and softmax, and significantly reduce the cost of secure inference while preserving the model performance. Additionally, we design secure Embedding and LayerNorm procedures that faithfully implement the desired functionality without undermining the Transformer architecture. PUMA is about $2\times$ faster than the state-of-the-art framework MPCFORMER(ICLR 2023) and has similar accuracy as plaintext models without fine-tuning (which the previous works failed to achieve). PUMA can even evaluate LLaMA-7B in around 5 minutes to generate 1 token. To our best knowledge, this is the first time that a model with such a parameter size is able to be evaluated under MPC. PUMA has been open-sourced in the Github repository of SecretFlow-SPU.
Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- MERGE: Fast Private Text GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Privformer: Privacy-preserving Transformer with MPC2023
- Characterization of MPC-based Private Inference for Transformer-based Models2022
- CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU2021
- Falcon: Honest-Majority Maliciously Secure Framework for Private Deep LearningProceedings on Privacy Enhancing Technologies, 2020
- CrypTFlow2: Practical 2-Party Secure Inference2020
- CrypTFlow: Secure TensorFlow Inference2020
- BLAZE: Blazing Fast Privacy-Preserving Machine Learning2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- ABY 32018