Limitations of Autoregressive Models and Their Alternatives
Authors:
- Chu-Cheng LinCLChu-Cheng Lin
- Aaron JaechAJAaron Jaech
- Xin LiXLXin Li
- Matthew R. GormleyMGMatthew R. Gormley
- Jason EisnerJEJason Eisner
Preprint
Abstract
Standard autoregressive language models perform only polynomial-time computation to compute the probability of the next symbol. While this is attractive, it means they cannot model distributions whose next-symbol probability is hard to compute. Indeed, they cannot even model them well enough to solve associated easy decision problems for which an engineer might want to consult a language model. These limitations apply no matter how much computation and data are used to train the model, unless the model is given access to oracle parameters that grow superpolynomially in sequence length. Thus, simply training larger autoregressive language models is not a panacea for NLP. Alternatives include energy-based models (which give up efficient sampling) and latent-variable autoregressive models (which give up efficient scoring of a given string). Both are powerful enough to escape the above limitations.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- On the Dangers of Stochastic Parrots2021
- Whole Sentence Neural Language Models2018
- Whole-sentence exponential language models: a vehicle for linguistic-statistical integrationComputer Speech & Language, 2001
- Approximate nearest neighbors1998
- On the hardness of approximate reasoningArtificial Intelligence, 1996
- On the computational power of neural nets1992
- Average Case Complete ProblemsSIAM Journal on Computing, 1986
- Some connections between nonuniform and uniform complexity classes1980
- The circuit value problem is log space complete for PACM SIGACT News, 1975
- The complexity of theorem-proving procedures1971