Transfer Learning for Text Diffusion Models
Preprint
Abstract
In this report, we explore the potential for text diffusion to replace autoregressive (AR) decoding for the training and deployment of large language models (LLMs). We are particularly interested to see whether pretrained AR models can be transformed into text diffusion models through a lightweight adaptation procedure we call ``AR2Diff''. We begin by establishing a strong baseline setup for training text diffusion models. Comparing across multiple architectures and pretraining objectives, we find that training a decoder-only model with a prefix LM objective is best or near-best across several tasks. Building on this finding, we test various transfer learning setups for text diffusion models. On machine translation, we find that text diffusion underperforms the standard AR approach. However, on code synthesis and extractive QA, we find diffusion models trained from scratch outperform AR models in many cases. We also observe quality gains from AR2Diff -- adapting AR models to use diffusion decoding. These results are promising given that text diffusion is relatively underexplored and can be significantly faster than AR decoding for long text generation.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade2021
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer2021
- Non-Autoregressive Machine Translation with Latent Alignments2020
- Mask-Predict: Parallel Decoding of Conditional Masked Language Models2019
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement2018
- Sequence-Level Knowledge Distillation2016
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Findings of the 2014 Workshop on Statistical Machine Translation2014