Continuous Visual Autoregressive Generation via Score Maximization
Preprint
Abstract
Conventional wisdom suggests that autoregressive models are used to process discrete data. When applied to continuous modalities such as visual data, Visual AutoRegressive modeling (VAR) typically resorts to quantization-based approaches to cast the data into a discrete space, which can introduce significant information loss. To tackle this issue, we introduce a Continuous VAR framework that enables direct visual autoregressive generation without vector quantization. The underlying theoretical foundation is strictly proper scoring rules, which provide powerful statistical tools capable of evaluating how well a generative model approximates the true distribution. Within this framework, all we need is to select a strictly proper score and set it as the training objective to optimize. We primarily explore a class of training objectives based on the energy score, which is likelihood-free and thus overcomes the difficulty of making probabilistic predictions in the continuous space. Previous efforts on continuous autoregressive generation, such as GIVT and diffusion loss, can also be derived from our framework using other strictly proper scores. Source code: https://github.com/shaochenze/EAR.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learningNeural Networks, 2018
- Universal Language Model Fine-tuning for Text Classification2018
- Layer-Specific Adaptive Learning Rates for Deep Networks2015
- Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*Monthly Weather Review, 2015
- Energy statistics: A class of statistics based on distancesJournal of Statistical Planning and Inference, 2013
- ImageNet: A large-scale hierarchical image database2009
- Strictly Proper Scoring Rules, Prediction, and EstimationJournal of the American Statistical Association, 2007
- Revisiting squared-error and cross-entropy functions for training neural network classifiersNeural Computing and Applications, 2005
- A note on least-squares learning procedures and classification by neural network modelsIEEE Transactions on Neural Networks, 1991
- Scoring Rules for Continuous Probability DistributionsManagement Science, 1976