EAEmily J. Allen

Emily J. Allen

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Emily J. Allen has a total of 42 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Processing of natural scenes in the human pulvinar

    by Daniel R. Guest, Emily J. Allen, Kendrick N. Kay, Michael J. Arcaro

    Theories of high-level visual processing and object recognition have typically focused on ventral visual cortex, often overlooking potential contributions of subcortical structures. The pulvinar, via extensive connections with visual cortex, is well-positioned to play a role in high-level vision. Here, we investigated how the pulvinar represents visual information using a high-resolution 7 T fMRI dataset of responses to tens of thousands of natural scenes. Encoding models targeting different stimulus features revealed a pulvinar region selective for bodies and faces presented in the contralateral visual hemifield. Complementary model-free analyses demonstrated that this region is co-active with body- and face-selective cortical areas during natural scene viewing. These findings challenge cortico-centric models of object vision by demonstrating that principles of cortical organization, including functional clustering and hierarchical organization, also manifest in subcortex. Moreover, they highlight the value of using naturalistic stimuli to probe visual function.


  • Top Cited Papers
    Open Access

    High-level visual representations in the human brain are aligned with large language models

    by Adrien Doerig, Tim C. Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, Ian Charest

    The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment. However, a quantitative approach for studying this information remains elusive. Here we test whether the contextual information encoded in large language models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes. We show that LLM embeddings of scene captions successfully characterize brain activity evoked by viewing the natural scenes. This mapping captures selectivities of different brain areas and is sufficiently robust that accurate scene captions can be reconstructed from brain activity. Using carefully controlled model comparisons, we then proceed to show that the accuracy with which LLM representations match brain representations derives from the ability of LLMs to integrate complex information contained in scene captions beyond that conveyed by individual words. Finally, we train deep neural network models to transform image inputs into LLM representations. Remarkably, these networks learn representations that are better aligned with brain representations than a large number of state-of-the-art alternative models, despite being trained on orders-of-magnitude less data. Overall, our results suggest that LLM embeddings of scene captions provide a representational format that accounts for complex information extracted by the brain from visual inputs.


  • Open Access

    Temporal asymmetry of neural pattern similarity predicts recognition memory decisions

    by Zhifang Ye, Yufei Zhao, Emily J. Allen, Thomas Naselaris, Kendrick Kay, J. Benjamin Hutchinson, Brice A. Kuhl

    A stimulus can be familiar for multiple reasons. It might have been recently encountered, is similar to recent experience, or is similar to ‘typical’ experience. Understanding how the brain translates these sources of similarity into memory decisions is a fundamental, but challenging goal. Here, using functional magnetic resonance imaging (fMRI), we computed neural similarity between a current stimulus and events from different temporal windows in the past and future (from seconds to days). We show that trial-by-trial recognition memory decisions (is this stimulus ‘old’?) were predicted by the difference in similarity to past vs. future events (temporal asymmetry). This relationship was (i) evident in lateral parietal and occipitotemporal cortices, (ii) strongest when considering events from the recent past (minutes ago), and (iii) most pronounced when veridical (true) memories were weak. These findings elucidate how the brain evaluates past experience in service of making recognition memory decisions.

  • Open Access

    Cortical substrates of perceptual confusion between pitch and timbre

    by Yongtian Ou, Emily J. Allen, Kendrick N. Kay, Andrew J. Oxenham

    Pitch and timbre are two fundamental perceptual attributes of sound that help us distinguish voices in speech and appreciate music. Brightness, one of the primary dimensions of timbre, is governed by different acoustic parameters compared to pitch, but the two can be confused perceptually when varied simultaneously. Here we combine human behavior and fMRI to provide evidence of a potential neural substrate to explain this important but poorly understood perceptual confusion. We identify orderly mappings of both pitch and brightness within auditory cortex and reveal two independent lines of evidence for cortical confusion between them. First, the preferred pitch of individual voxels decreases systematically as brightness increases, and vice versa, consistent with predictions based on perceptual confusion. Second, pitch and brightness mapping share a common high-low-high gradient across auditory cortex, implying a shared trajectory of cortical activation for changes in each dimension. The results provide a cortical substrate at both local and global scales for an established auditory perceptual phenomenon that is thought to reflect efficient coding of features ubiquitous in natural sound statistics.


  • Open Access

    Re-expression of CA1 and entorhinal activity patterns preserves temporal context memory at long timescales

    by Futing Zou, Guo Wanjia, Emily J. Allen, Yihan Wu, Ian Charest, Thomas Naselaris, Kendrick Kay, Brice A. Kuhl, J. Benjamin Hutchinson, Sarah DuBrow

    Converging, cross-species evidence indicates that memory for time is supported by hippocampal area CA1 and entorhinal cortex. However, limited evidence characterizes how these regions preserve temporal memories over long timescales (e.g., months). At long timescales, memoranda may be encountered in multiple temporal contexts, potentially creating interference. Here, using 7T fMRI, we measured CA1 and entorhinal activity patterns as human participants viewed thousands of natural scene images distributed, and repeated, across many months. We show that memory for an image’s original temporal context was predicted by the degree to which CA1/entorhinal activity patterns from the first encounter with an image were re-expressed during re-encounters occurring minutes to months later. Critically, temporal memory signals were dissociable from predictors of recognition confidence, which were carried by distinct medial temporal lobe expressions. These findings suggest that CA1 and entorhinal cortex preserve temporal memories across long timescales by coding for and reinstating temporal context information.


  • Open Access

    Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations

    by Ghislain St-Yves, Emily J. Allen, Yihan Wu, Kendrick Kay, Thomas Naselaris

    Deep neural networks (DNNs) optimized for visual tasks learn representations that align layer depth with the hierarchy of visual areas in the primate brain. One interpretation of this finding is that hierarchical representations are necessary to accurately predict brain activity in the primate visual system. To test this interpretation, we optimized DNNs to directly predict brain activity measured with fMRI in human visual areas V1-V4. We trained a single-branch DNN to predict activity in all four visual areas jointly, and a multi-branch DNN to predict each visual area independently. Although it was possible for the multi-branch DNN to learn hierarchical representations, only the single-branch DNN did so. This result shows that hierarchical representations are not necessary to accurately predict human brain activity in V1-V4, and that DNNs that encode brain-like visual representations may differ widely in their architecture, ranging from strict serial hierarchies to multiple independent branches.


  • Open Access

    Color-biased regions in the ventral visual pathway are food-selective

    by Ian Morgan Leo Pennock, Chris Racey, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, Anna Franklin, Jenny Bosten

    The ventral visual pathway is well known to be involved in recognizing and categorizing objects (Kanwisher and Dilks, 2013). Three color-biased areas have also been found between face and place selective areas in the ventral visual pathway (Lafer-Sousa et al., 2016). To understand the function of these color-biased areas in a region known for object recognition, we analyzed the Natural Scenes Dataset (NSD; Allen et al., 2022), a large 7T fMRI dataset from 8 participants who viewed up to 30,000 trials of images of colored natural scenes. In a whole-brain analysis, we correlated the average color saturation of the images and the voxel responses, revealing color-biased areas that diverge into two streams in the ventral visual pathway, beginning in V4 and extending medially and laterally of the Fusiform Face Area in both hemispheres. We drew regions of interest (ROIs) for the two streams and found that the images for each ROI that evoked the largest responses had certain characteristics: They contained food, contained circular objects, had higher color saturation, contained warmer hues, and had more luminance entropy. A multiple linear regression showed that the presence of food in images was the strongest predictor of voxel responses in the medial and lateral color-biased regions for all eight participants, but that color saturation also contributed independently to voxel responses. Our results show that these areas are food-selective and color biased. We suggest that these streams might be involved in using color to recognize and judge the properties of food.


  • Top Cited Papers

    A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

    by Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al.

    Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. Here we present the Natural Scenes Dataset (NSD), in which high-resolution functional magnetic resonance imaging responses to tens of thousands of richly annotated natural scenes were measured while participants performed a continuous recognition task. To optimize data quality, we developed and applied novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we used NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality and breadth, NSD opens new avenues of inquiry in cognitive neuroscience and artificial intelligence.


  • Open Access

    Distinct Representations of Tonotopy and Pitch in Human Auditory Cortex

    by Emily J. Allen, Juraj Mesik, Kendrick N. Kay, Andrew J. Oxenham

    Frequency-to-place mapping, or tonotopy, is a fundamental organizing principle throughout the auditory system, from the earliest stages of auditory processing in the cochlea to subcortical and cortical regions. Although cortical maps are referred to as tonotopic, it is unclear whether they simply reflect a mapping of physical frequency inherited from the cochlea, a computation of pitch based on the fundamental frequency, or a mixture of these two features. We used high-resolution functional magnetic resonance imaging (fMRI) to measure BOLD responses as male and female human participants listened to pure tones that varied in frequency or complex tones that varied in either spectral content (brightness) or fundamental frequency (pitch). Our results reveal evidence for pitch tuning in bilateral regions that partially overlap with the traditional tonotopic maps of spectral content. In general, primary regions within Heschl's gyri (HGs) exhibited more tuning to spectral content, whereas areas surrounding HGs exhibited more tuning to pitch.SIGNIFICANCE STATEMENTTonotopy, an orderly mapping of frequency, is observed throughout the auditory system. However, it is not known whether the tonotopy observed in the cortex simply reflects the frequency spectrum (as in the ear) or instead represents the higher-level feature of fundamental frequency, or pitch. Using carefully controlled stimuli and high-resolution functional magnetic resonance imaging (fMRI), we separated these features to study their cortical representations. Our results suggest that tonotopy in primary cortical regions is driven predominantly by frequency, but also reveal evidence for tuning to pitch in regions that partially overlap with the tonotopic gradients but extend into nonprimary cortical areas. In addition to resolving ambiguities surrounding cortical tonotopy, our findings provide evidence that selectivity for pitch is distributed bilaterally throughout auditory cortex.


  • Open Access

    The Natural Scenes Dataset (NSD): A yearlong ultra-high field whole-brain human fMRI visual perception and memory study

    by Emily J. Allen, Yihan Wu, J. Benjamin Hutchinson, Thomas Naselaris, Kendrick N. Kay

    Developing accurate, generalizable models of visual representation requires extensive stimulus sampling due to the vast dimensionality of visual inputs. The aim of this study was to design, collect, and pre-process a massive, high-quality fMRI dataset that can be used to advance our understanding of visual representation in the human brain. Using ultra-high-field fMRI (7T, whole-brain, T2*-weighted gradient-echo EPI, 1.8-mm resolution, 1.6-s TR), we measured BOLD responses while each of 8 participants viewed 9,000–10,000 distinct, color natural scenes (22,500–30,000 trials) in 30–40 weekly scan sessions over the course of a year. As participants fixated a central point, they performed a long-term continuous recognition task in which they judged whether they had seen each image at any time during the experiment, either in the current scan session or any previous scan session. Data collection is now complete and includes functional data during the continuous recognition task, as well as resting-state data, retinotopy, category localizers, anatomical data (T1, T2, diffusion, venogram, angiogram), physiological data (pulse, respiration), eye-tracking data, and additional behavioral assessments outside the scanner. Pre-processing and data quality assessments indicate that the data are of exceptional quality, with participants having nearly perfect response rates, high task performance, and low head motion, and with brain images having high contrast-to-noise ratio and spatial stability across scan sessions. Both the raw and pre-processed data will be made publicly available to the scientific community. Because of its unprecedented scale and richness, NSD (http://naturalscenesdataset.org) can be used to explore diverse neuroscientific questions with high power at the level of individual subjects and to develop models of the complex series of transformations occurring throughout the visual hierarchy.