John J. Hopfield
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
John J. Hopfield has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Dmitry Krotov
MIT IBM Watson AI Lab IBM Research Princeton Neuroscience Institute Princeton University
4 shared publications Carlos D. Brody
Departments of *Molecular Biology and
3 shared publications- Dmitry Krotov
Institute for Advanced Study, Princeton, NJ 08540, U.S.A.
3 shared publications - Carlos D. Brody
†Department of Molecular Biology, Princeton University, Princeton, NJ 08544-1014; and ¶Cold Spring Harbor Laboratory, P.O. Box 100, Cold Spring Harbor, NY 11724
1 shared publication D. W. Tank
Molecular Biophysics Research Department, AT&T Bell Laboratories, Murray Hill, NJ 07974 USA
1 shared publicationDavid A. Markowitz
Departments of Molecular Biology and Physics, The Lewis Sigler Institute for Integrative Genomics, and Princeton Neuroscience Institute, Carl Icahn Laboratory, Princeton University, Princeton, NJ 08544, USA
1 shared publication
Latest Publications
- Open Access
Large Associative Memory Problem in Neurobiology and Machine Learning
Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.
Open AccessLarge Associative Memory Problem in Neurobiology and Machine Learning
Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.
- Open Access
Unsupervised Learning by Competing Hidden Units
It is widely believed that the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility, and which is motivated by Hebb's idea that change of the synapse strength should be local - i.e. should depend only on the activities of the pre and post synaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer, and is capable of learning early feature detectors in a completely unsupervised way. These learned lower layer feature detectors can be used to train higher layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm.
Open AccessUnsupervised Learning by Competing Hidden Units
It is widely believed that the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility, and which is motivated by Hebb's idea that change of the synapse strength should be local - i.e. should depend only on the activities of the pre and post synaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer, and is capable of learning early feature detectors in a completely unsupervised way. These learned lower layer feature detectors can be used to train higher layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm.
- Open Access
Local Unsupervised Learning for Image Analysis
Local Hebbian learning is believed to be inferior in performance to end-to-end training using a backpropagation algorithm. We question this popular belief by designing a local algorithm that can learn convolutional filters at scale on large image datasets. These filters combined with patch normalization and very steep non-linearities result in a good classification accuracy for shallow networks trained locally, as opposed to end-to-end. The filters learned by our algorithm contain both orientation selective units and unoriented color units, resembling the responses of pyramidal neurons located in the cytochrome oxidase 'interblob' and 'blob' regions in the primary visual cortex of primates. It is shown that convolutional networks with patch normalization significantly outperform standard convolutional networks on the task of recovering the original classes when shadows are superimposed on top of standard CIFAR-10 images. Patch normalization approximates the retinal adaptation to the mean light intensity, important for human vision. We also demonstrate a successful transfer of learned representations between CIFAR-10 and ImageNet 32x32 datasets. All these results taken together hint at the possibility that local unsupervised training might be a powerful tool for learning general representations (without specifying the task) directly from unlabeled data.
Open AccessLocal Unsupervised Learning for Image Analysis
Local Hebbian learning is believed to be inferior in performance to end-to-end training using a backpropagation algorithm. We question this popular belief by designing a local algorithm that can learn convolutional filters at scale on large image datasets. These filters combined with patch normalization and very steep non-linearities result in a good classification accuracy for shallow networks trained locally, as opposed to end-to-end. The filters learned by our algorithm contain both orientation selective units and unoriented color units, resembling the responses of pyramidal neurons located in the cytochrome oxidase 'interblob' and 'blob' regions in the primary visual cortex of primates. It is shown that convolutional networks with patch normalization significantly outperform standard convolutional networks on the task of recovering the original classes when shadows are superimposed on top of standard CIFAR-10 images. Patch normalization approximates the retinal adaptation to the mean light intensity, important for human vision. We also demonstrate a successful transfer of learned representations between CIFAR-10 and ImageNet 32x32 datasets. All these results taken together hint at the possibility that local unsupervised training might be a powerful tool for learning general representations (without specifying the task) directly from unlabeled data.
- Open Access
- 29 March 2019
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 116 (16) , 7723-7731
Unsupervised learning by competing hidden units
It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb’s idea that change of the synapse strength should be local—i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.
- 29 March 2019
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 116 (16) , 7723-7731
Open AccessUnsupervised learning by competing hidden units
It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb’s idea that change of the synapse strength should be local—i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.
- 1 December 2018
- Published by MIT Press in Neural Computation
- Vol. 30 (12) , 3151-3167
Dense Associative Memory Is Robust to Adversarial Inputs
Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNNs and humans classify patterns and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our article examines these questions within the framework of dense associative memory (DAM) models. These models are defined by the energy function, with higher-order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units, fail to transfer to and fool the models with higher-order interactions. This opens up the possibility of using higher-order models for detecting and stopping malicious adversarial attacks. The results we present suggest that DAMs with higher-order energy functions are more robust to adversarial and rubbish inputs than DNNs with rectified linear units.
- 1 December 2018
- Published by MIT Press in Neural Computation
- Vol. 30 (12) , 3151-3167
Dense Associative Memory Is Robust to Adversarial Inputs
Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNNs and humans classify patterns and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our article examines these questions within the framework of dense associative memory (DAM) models. These models are defined by the energy function, with higher-order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units, fail to transfer to and fool the models with higher-order interactions. This opens up the possibility of using higher-order models for detecting and stopping malicious adversarial attacks. The results we present suggest that DAMs with higher-order energy functions are more robust to adversarial and rubbish inputs than DNNs with rectified linear units.
- Open Access
Dense Associative Memory is Robust to Adversarial Inputs
Deep neural networks (DNN) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation, so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNN and humans classify patterns, and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our paper examines these questions within the framework of Dense Associative Memory (DAM) models. These models are defined by the energy function, with higher order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units (ReLU), fail to transfer to and fool the models with higher order interactions. This opens up a possibility to use higher order models for detecting and stopping malicious adversarial attacks. The presented results suggest that DAM with higher order energy functions are closer to human visual perception than DNN with ReLUs.
Open AccessDense Associative Memory is Robust to Adversarial Inputs
Deep neural networks (DNN) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish examples or fooling images) that lack semantic similarity with the training data. Second, a clean input can be changed by a small, and often imperceptible for human vision, perturbation, so that the resulting deformed input is misclassified by the network. These findings emphasize the differences between the ways DNN and humans classify patterns, and raise a question of designing learning algorithms that more accurately mimic human perception compared to the existing methods. Our paper examines these questions within the framework of Dense Associative Memory (DAM) models. These models are defined by the energy function, with higher order (higher than quadratic) interactions between the neurons. We show that in the limit when the power of the interaction vertex in the energy function is sufficiently large, these models have the following three properties. First, the minima of the objective function are free from rubbish images, so that each minimum is a semantically meaningful pattern. Second, artificial patterns poised precisely at the decision boundary look ambiguous to human subjects and share aspects of both classes that are separated by that decision boundary. Third, adversarial images constructed by models with small power of the interaction vertex, which are equivalent to DNN with rectified linear units (ReLU), fail to transfer to and fool the models with higher order interactions. This opens up a possibility to use higher order models for detecting and stopping malicious adversarial attacks. The presented results suggest that DAM with higher order energy functions are closer to human visual perception than DNN with ReLUs.
- Open Access
Dense Associative Memory for Pattern Recognition
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.
Open AccessDense Associative Memory for Pattern Recognition
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.
- 1 October 2015
- Published by MIT Press in Neural Computation
- Vol. 27 (10) , 2011-2038
Understanding Emergent Dynamics: Using a Collective Activity Coordinate of a Neural Network to Recognize Time-Varying Patterns
In higher animals, complex and robust behaviors are produced by the microscopic details of large structured ensembles of neurons. I describe how the emergent computational dynamics of a biologically based neural network generates a robust natural solution to the problem of categorizing time-varying stimulus patterns such as spoken words or animal stereotypical behaviors. The recognition of these patterns is made difficult by their substantial variation in cadence and duration. The neural circuit behaviors used are similar to those associated with brain neural integrators. In the larger context described here, this kind of circuit becomes a building block of an entirely different computational algorithm for solving complex problems. While the network behavior is simulated in detail, a collective view is essential to understanding the results. A closed equation of motion for the collective variable describes an algorithm that quantitatively accounts for many aspects of the emergent network computation. The feedback connections and ongoing activity in the network shape the collective dynamics onto a reduced dimensionality manifold of activity space, which defines the algorithm and computation actually performed. The external inputs are weak and are not the dominant drivers of network activity.
- 1 October 2015
- Published by MIT Press in Neural Computation
- Vol. 27 (10) , 2011-2038
Understanding Emergent Dynamics: Using a Collective Activity Coordinate of a Neural Network to Recognize Time-Varying Patterns
In higher animals, complex and robust behaviors are produced by the microscopic details of large structured ensembles of neurons. I describe how the emergent computational dynamics of a biologically based neural network generates a robust natural solution to the problem of categorizing time-varying stimulus patterns such as spoken words or animal stereotypical behaviors. The recognition of these patterns is made difficult by their substantial variation in cadence and duration. The neural circuit behaviors used are similar to those associated with brain neural integrators. In the larger context described here, this kind of circuit becomes a building block of an entirely different computational algorithm for solving complex problems. While the network behavior is simulated in detail, a collective view is essential to understanding the results. A closed equation of motion for the collective variable describes an algorithm that quantitatively accounts for many aspects of the emergent network computation. The feedback connections and ongoing activity in the network shape the collective dynamics onto a reduced dimensionality manifold of activity space, which defines the algorithm and computation actually performed. The external inputs are weak and are not the dominant drivers of network activity.
- 30 December 2009
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 107 (4) , 1648-1653
Neurodynamics of mental exploration
Thinking allows an animal to take an effective action in a novel situation based on a mental exploration of possibilities and previous knowledge. We describe a model animal, with a neural system based loosely on the rodent hippocampus, which performs mental exploration to find a useful route in a spatial world it has previously learned. It then mentally recapitulates the chosen route, and this intent is converted to motor acts that move the animal physically along the route. The modeling is based on spiking neurons with spike-frequency adaptation. Adaptation causes the continuing evolution in the pattern of neural activity that is essential to mental exploration. A successful mental exploration is remembered through spike-timing-dependent synaptic plasticity. The system is also an episodic memory for an animal chiefly concerned with locations.
- 30 December 2009
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 107 (4) , 1648-1653
Neurodynamics of mental exploration
Thinking allows an animal to take an effective action in a novel situation based on a mental exploration of possibilities and previous knowledge. We describe a model animal, with a neural system based loosely on the rodent hippocampus, which performs mental exploration to find a useful route in a spatial world it has previously learned. It then mentally recapitulates the chosen route, and this intent is converted to motor acts that move the animal physically along the route. The modeling is based on spiking neurons with spike-frequency adaptation. Adaptation causes the continuing evolution in the pattern of neural activity that is essential to mental exploration. A successful mental exploration is remembered through spike-timing-dependent synaptic plasticity. The system is also an episodic memory for an animal chiefly concerned with locations.
- 17 June 2008
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 105 (24) , 8422-8427
Rate-specific synchrony: Using noisy oscillations to detect equally active neurons
Although gamma frequency oscillations are common in the brain, their functional contributions to neural computation are not understood. Here we report in vitro electrophysiological recordings to evaluate how noisy gamma frequency oscillatory input interacts with the overall activation level of a neuron to determine the precise timing of its action potentials. The experiments were designed to evaluate spike synchrony in a neural circuit architecture in which a population of neurons receives a common noisy gamma oscillatory synaptic drive while the firing rate of each individual neuron is determined by a slowly varying independent input. We demonstrate that similarity of firing rate is a major determinant of synchrony under common noisy oscillatory input: Near coincidence of spikes at similar rates gives way to substantial desynchronization at larger firing rate differences. Analysis of this rate-specific synchrony phenomenon reveals distinct spike timing "fingerprints" at different firing rates that emerge through a combination of phase shifting and abrupt changes in spike patterns. We further demonstrate that rate-specific synchrony permits robust detection of rate similarity in a population of neurons through synchronous activation of a postsynaptic neuron, supporting the biological plausibility of a Many Are Equal computation. Our results reveal that spatially coherent noisy oscillations, which are common throughout the brain, can generate previously unknown relationships among neural rate codes, noisy interspike intervals, and precise spike synchrony codes. All of these can coexist in a self-consistent manner because of rate-specific synchrony.
- 17 June 2008
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 105 (24) , 8422-8427
Rate-specific synchrony: Using noisy oscillations to detect equally active neurons
Although gamma frequency oscillations are common in the brain, their functional contributions to neural computation are not understood. Here we report in vitro electrophysiological recordings to evaluate how noisy gamma frequency oscillatory input interacts with the overall activation level of a neuron to determine the precise timing of its action potentials. The experiments were designed to evaluate spike synchrony in a neural circuit architecture in which a population of neurons receives a common noisy gamma oscillatory synaptic drive while the firing rate of each individual neuron is determined by a slowly varying independent input. We demonstrate that similarity of firing rate is a major determinant of synchrony under common noisy oscillatory input: Near coincidence of spikes at similar rates gives way to substantial desynchronization at larger firing rate differences. Analysis of this rate-specific synchrony phenomenon reveals distinct spike timing "fingerprints" at different firing rates that emerge through a combination of phase shifting and abrupt changes in spike patterns. We further demonstrate that rate-specific synchrony permits robust detection of rate similarity in a population of neurons through synchronous activation of a postsynaptic neuron, supporting the biological plausibility of a Many Are Equal computation. Our results reveal that spatially coherent noisy oscillations, which are common throughout the brain, can generate previously unknown relationships among neural rate codes, noisy interspike intervals, and precise spike synchrony codes. All of these can coexist in a self-consistent manner because of rate-specific synchrony.