Attention in a Family of Boltzmann Machines Emerging From Modern Hopfield Networks
- 12 July 2023
- journal article
- research article
- Published by MIT Press in Neural Computation
- Vol. 35 (8) , 1463-1480
Abstract
Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broadened the class of energy functions and led to a unified perspective on general Hopfield networks, including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as the attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for certain special cases and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the gaussian–Bernoulli restricted BM and the denoising autoencoder with softmax units coming from denoising score matching. We also investigate BMs introduced by other energy functions and show that the energy function of dense associative memory models gives BMs belonging to exponential family harmoniums.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Modern Hopfield Networks and Attention for Immune Repertoire ClassificationbioRxiv, 2020
- On a Model of Associative Memory with Huge Storage CapacityJournal of Statistical Physics, 2017
- Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible unitsNeural Networks, 2016
- A Connection Between Score Matching and Denoising AutoencodersNeural Computation, 2011
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Neurons with graded response have collective computational properties like those of two-state neurons.Proceedings of the National Academy of Sciences of the United States of America, 1984
- Neural networks and physical systems with emergent collective computational abilities.Proceedings of the National Academy of Sciences of the United States of America, 1982