Frank Tong

Deep neural networks (DNNs) offer highly promising neurocomputational models of the visual system, yet vast gaps remain between DNNs and human observers. By some accounts, DNNs are approaching near ceiling levels in their ability to predict human neural responses to clear real-world images. However, even modest diversions toward more ambiguous viewing conditions can readily expose the brittle and inflexible nature of these networks. Human vision remains robust when faced with noise, blur, occlusion, and other challenges, whereas DNNs trained to classify large image datasets typically lack such robustness. Here, we discuss the ecological vision hypothesis, proposing that the robustness of human vision is acquired via learning from prevalent encounters with challenging viewing conditions, such that DNNs trained with similar challenges should become more robust and human-aligned. In particular, the prevalence of blur in everyday vision may enhance sensitivity to global shape and attenuate reliance on local textural cues. We conjecture that providing DNNs with ecologically relevant information to learn 3D scene and shape properties will further advance DNN-to-human alignment.

Human object recognition is robust to challenging conditions, such as when one's view of an object is fragmented due to an occluding foreground object. In comparison, deep neural networks (DNNs) are typically more susceptible to occlusion, suggesting that human vision relies on distinct mechanisms. Here, we investigated the role of visual diet in the emergence of these mechanisms by asking whether human-like robustness might arise in DNNs when trained with image datasets that better reflect the properties of occlusion in natural vision. We trained convolutional and transformer DNNs to classify clear images only, images augmented with artificial occluders (i.e., geometric shapes) or natural occluders (objects segmented from photographs). We then evaluated DNN occlusion robustness and compared their performance profiles with 30 human participants. We found that DNNs trained with artificial occluders remained vulnerable to natural occlusion and exhibited less human-like performance than those trained with natural occlusion. Our findings suggest that human robustness to visual occlusion arises from learning to disentangle natural objects from each other rather than simply learning to recognize objects from partial views. They also imply that commonly used forms of artificial occlusion are unsuitable for the evaluation or promotion of robustness to real-world occlusion in DNNs.

A fundamental challenge in motion perception lies in the fact that the global motion of any translating object will generate a heterogeneous set of local motion signals that vary depending on the oriented contour information within each local region. This so-called aperture problem illustrates how the visual system must integrate diverse types of motion signals to attain direction selectivity that is invariant to visual form. Here, we investigated how form-invariant motion selectivity emerges across the human visual pathway by using functional magnetic resonance imaging (fMRI) to measure direction-selective responses to drifting gratings and random-dot motion, and then testing for reliable generalization across stimulus types. All visual areas of interest showed highly reliable direction-selective classification performance for a given stimulus type, but early areas V1 and V2 showed chance-level generalization between motion types. Indeed, V1 responses tended to confuse random-dot motion as resembling orthogonal grating motion, implying that drifting random dots generated oriented motion-streak responses in V1. By contrast, motion-sensitive areas MT+ and V3A showed reliable cross-generalization performance in our fMRI experiments that tested both linear and spiral motion trajectories. Our findings provide compelling evidence that motion direction selectivity becomes more invariant to stimulus form in higher visual areas, particularly along the dorsal visual pathway.

The constructive nature of motion perception has been highlighted in studies of the visual phantom illusion. Visual phantoms can occur when two low-contrast collinear drifting gratings are separated by a blank gap, leading to the ghostly impression of drifting stripes that extend through the gap. Although previous work has shown that phantom-inducing gratings can elicit a motion aftereffect (MAE) in the gap region, it is not known whether these MAEs arise from the perception of visual phantoms per se. Here, we evaluated the strength of MAEs elicited by phantom-inducing gratings and well-matched control stimuli. Either the darkest portion of the inducer gratings matched the background luminance to elicit more vivid phantoms or the mean luminance of the inducers matched the background as a control. Both phantom and control inducers predominantly evoked impressions of opposite-direction motion, but the dynamic MAE proved somewhat stronger for phantom than control inducers. Also, MAEs were more strongly modulated by the physical contrast of the control inducers and less influenced by the contrast of the phantom inducers. These findings suggest that low-level motion adaptation strongly contributes to the MAEs elicited in the gap region but that the magnitude of adaptation is further modulated by phantom perception.

Object recognition in real-world environments requires dealing with considerable ambiguity, yet the human visual system is highly robust to noisy viewing conditions. Here, we investigated the role of perceptual learning in the acquisition of robustness in both humans and deep neural networks (DNNs). Specifically, we sought to determine whether perceptual training with object images in Gaussian noise, drawn from certain animate or inanimate categories, would lead to category-specific or category-general improvements in human robustness. Moreover, might DNNs provide viable models of human perceptual learning? Both before and after training, we evaluated the noise threshold required for accurate recognition using novel object images. Human observers were quite robust to noise before training, but showed additional category-specific improvement after training with only a few hundred noisy object examples. In comparison, standard DNNs initially lacked robustness, then showed both category-general and category-specific learning after training with the same noisy examples. We further evaluated DNN models that were pre-trained with moderately noisy images to match human pre-training accuracy. Notably, these models only showed category-specific improvement, matching the overall pattern of learning exhibited by human observers. A layer-wise analysis of DNN responses revealed that category-general learning effects emerged in the lower layers, whereas category-specific improvements emerged in the higher layers. Our findings provide support for the notion that robustness to noisy visual conditions arises through learning, humans likely acquire robustness from everyday encounters with real-world noise, and additional category-specific improvements exhibited by humans and DNNs involve learning at higher levels of visual representation.

The editorial introduces the Special Section on Medical Image Perception and Observer Performance for JMI Volume 12 Issue 5.

Computational models of the primary visual cortex (V1) have suggested that V1 neurons behave like Gabor filters followed by simple nonlinearities. However, recent work employing convolutional neural network (CNN) models has suggested that V1 relies on far more nonlinear computations than previously thought. Specifically, unit responses in an intermediate layer of VGG-19 were found to best predict macaque V1 responses to thousands of natural and synthetic images. Here, we evaluated the hypothesis that the poor performance of lower layer units in VGG-19 might be attributable to their small receptive field size rather than to their lack of complexity per se. We compared VGG-19 with AlexNet, which has much larger receptive fields in its lower layers. Whereas the best-performing layer of VGG-19 occurred after seven nonlinear steps, the first convolutional layer of AlexNet best predicted V1 responses. Although the predictive accuracy of VGG-19 was somewhat better than that of standard AlexNet, we found that a modified version of AlexNet could match the performance of VGG-19 after only a few nonlinear computations. Control analyses revealed that decreasing the size of the input images caused the best-performing layer of VGG-19 to shift to a lower layer, consistent with the hypothesis that the relationship between image size and receptive field size can strongly affect model performance. We conducted additional analyses using a Gabor pyramid model to test for nonlinear contributions of normalization and contrast saturation. Overall, our findings suggest that the feedforward responses of V1 neurons can be well explained by assuming only a few nonlinear processing stages.

Whenever a visual scene is cast onto the retina, much of it will appear degraded due to poor resolution in the periphery; moreover, optical defocus can cause blur in central vision. However, the pervasiveness of blurry or degraded input is typically overlooked in the training of convolutional neural networks (CNNs). We hypothesized that the absence of blurry training inputs may cause CNNs to rely excessively on high spatial frequency information for object recognition, thereby causing systematic deviations from biological vision. We evaluated this hypothesis by comparing standard CNNs with CNNs trained on a combination of clear and blurry images. We show that blur-trained CNNs outperform standard CNNs at predicting neural responses to objects across a variety of viewing conditions. Moreover, blur-trained CNNs acquire increased sensitivity to shape information and greater robustness to multiple forms of visual noise, leading to improved correspondence with human perception. Our results provide multi-faceted neurocomputational evidence that blurry visual experiences may be critical for conferring robustness to biological visual systems.

Whenever a visual scene is cast onto the retina, much of it will appear degraded due to poor resolution in the periphery; moreover, optical defocus can cause blur in central vision. However, the pervasiveness of blurry or degraded input is typically overlooked in the training of convolutional neural networks (CNNs). We hypothesized that the absence of blurry training inputs may cause CNNs to rely excessively on high spatial frequency information for object recognition, thereby causing systematic deviations from biological vision. We evaluated this hypothesis by comparing standard CNNs with CNNs trained on a combination of clear and blurry images. We show that blur-trained CNNs outperform standard CNNs at predicting neural responses to objects across a variety of viewing conditions. Moreover, blur-trained CNNs acquire increased sensitivity to shape information and greater robustness to multiple forms of visual noise, leading to improved correspondence with human perception. Our results provide novel neurocomputational evidence that blurry visual experiences are very important for conferring robustness to biological visual systems.

Neural adaptation can be defined as reduction of spiking responses following repeated or prolonged stimulation. Adaptation helps adjust neural responsiveness to avoid saturation and has been suggested to improve perceptual selectivity, information transmission, and predictive coding. Here, we report rapid adaptation to repeated cycles of gratings drifting over the receptive field of neurons at the earliest site of postretinal processing, the lateral geniculate nucleus of the thalamus.