Regularized Vector Quantization for Tokenized Image Synthesis
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 18467-18476
Abstract
Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. Predominant approaches learn the discrete representation either in a deterministic manner by selecting the best-matching token or in a stochastic manner by sampling from a predicted distribution. However, deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while stochastic quantization suffers from low codebook utilization and perturbed reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate above issues effectively by applying regularization from two perspectives. The first is a prior distribution regularization which measures the discrepancy between a prior token distribution and the predicted token distribution to avoid code-book collapse and low codebook utilization. The second is a stochastic mask regularization that introduces stochasticity during quantization to strike a good balance between inference stage misalignment and unperturbed reconstruction objective. In addition, we design a probabilistic contrastive loss which serves as a calibrated metric to further mitigate the perturbed reconstruction objective. Extensive experiments show that the proposed quantization framework outperforms prevailing vector quantization methods consistently across different generative models including auto-regressive models and diffusion models.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Auto-regressive Image Synthesis with Integrated Quantization2022
- Taming Transformers for High-Resolution Image Synthesis2021
- Multimodal Machine Learning: A Survey and TaxonomyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Scene Parsing through ADE20K Dataset2017
- Deep Learning Face Attributes in the Wild2015
- Long Short-Term MemoryNeural Computation, 1997
- The expectation-maximization algorithmIEEE Signal Processing Magazine, 1996
- A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation AlgorithmsJournal of the American Statistical Association, 1990