Fraser W. Smith

Neurons in the primary visual cortex (V1) receive sensory inputs that describe small, local regions of the visual scene and cortical feedback inputs from higher visual areas processing the global scene context. Investigating the spatial precision of this visual contextual modulation will contribute to our understanding of the functional role of cortical feedback inputs in perceptual computations. We used human functional magnetic resonance imaging (fMRI) to test the spatial precision of contextual feedback inputs to V1 during natural scene processing. We measured brain activity patterns in the stimulated regions of V1 and in regions that we blocked from direct feedforward input, receiving information only from non-feedforward (i.e., feedback and lateral) inputs. We measured the spatial precision of contextual feedback signals by generalising brain activity patterns across parametrically spatially displaced versions of identical images using an MVPA cross-classification approach. We found that fMRI activity patterns in cortical feedback signals predicted our scene-specific features in V1 with a precision of approximately 4 degrees. The stimulated regions of V1 carried more precise scene information than non-stimulated regions; however, these regions also contained information patterns that generalised up to 4 degrees. This result shows that contextual signals relating to the global scene are similarly fed back to V1 when feedforward inputs are either present or absent. Our results are in line with contextual feedback signals from extrastriate areas to V1, describing global scene information and contributing to perceptual computations such as the hierarchical representation of feature boundaries within natural scenes.

Whilst previous research has linked attenuation of the mu rhythm to the observation of specific visual categories, and even to a potential role in action observation via a putative mirror neuron system, much of this work has not considered what specific type of information might be coded in this oscillatory response when triggered via vision. Here, we sought to determine whether the mu rhythm contains content-specific information about the identity of familiar (and also unfamiliar) graspable objects. In the present study, right-handed participants ( N =27) viewed images of both familiar (apple, wine glass) and unfamiliar (cubie, smoothie) graspable objects, whilst performing an orthogonal task at fixation. Multivariate pattern analysis (MVPA) revealed significant decoding of familiar, but not unfamiliar, visual object categories in the mu rhythm response. Thus, simply viewing familiar graspable objects may automatically trigger activation of associated tactile and/or motor properties in sensorimotor areas, reflected in the mu rhythm. In addition, we report significant attenuation in the central beta band for both familiar and unfamiliar visual objects, but not in the mu rhythm. Our findings highlight how analysing two different aspects of the oscillatory response – either attenuation or the representation of information content – provide complementary views on the role of the mu rhythm in response to viewing graspable object categories. Highlights: The Mu Rhythm oscillation contains fine-grained information about the identity of familiar, graspable objects (but not unfamiliar) This study offers evidence of a possible oscillatory marker for cross-sensory effects involving sensorimotor and visual cortices Different analysis techniques (univariate vs multivariate) imply different conclusions about the role of specific oscillations in the perception of graspable objects The alpha rhythm at occipital sites shows greater attenuation for unfamiliar objects but better representation for familiar objects consistent with sharpening accounts of Predictive Processing

Neurons, even in the earliest sensory regions of cortex, are subject to a great deal of contextual influences from both within and across modality connections. Recent work has shown that primary sensory areas can respond to and, in some cases, discriminate stimuli that are not of their target modality: for example, primary somatosensory cortex (SI) discriminates visual images of graspable objects. In the present work, we investigated whether SI would discriminate sounds depicting hand–object interactions (e.g. bouncing a ball). In a rapid event-related functional magnetic resonance imaging experiment, participants listened attentively to sounds from 3 categories: hand–object interactions, and control categories of pure tones and animal vocalizations, while performing a one-back repetition detection task. Multivoxel pattern analysis revealed significant decoding of hand–object interaction sounds within SI, but not for either control category. Crucially, in the hand-sensitive voxels defined from an independent tactile localizer, decoding accuracies were significantly higher for hand–object interactions compared to pure tones in left SI. Our findings indicate that simply hearing sounds depicting familiar hand–object interactions elicit different patterns of activity in SI, despite the complete absence of tactile stimulation. These results highlight the rich contextual information that can be transmitted across sensory modalities even to primary sensory areas.

Most neuroimaging experiments that investigate how tools and their actions are represented in the brain use visual paradigms where tools or hands are displayed as 2D images and no real movements are performed. These studies discovered selective visual responses in occipitotemporal and parietal cortices for viewing pictures of hands or tools, which are assumed to reflect action processing, but this has rarely been directly investigated. Here, we examined the responses of independently visually defined category-selective brain areas when participants grasped 3D tools (N = 20; 9 females). Using real-action fMRI and multivoxel pattern analysis, we found that grasp typicality representations (i.e., whether a tool is grasped appropriately for use) were decodable from hand-selective areas in occipitotemporal and parietal cortices, but not from tool-, object-, or body-selective areas, even if partially overlapping. Importantly, these effects were exclusive for actions with tools, but not for biomechanically matched actions with control nontools. In addition, grasp typicality decoding was significantly higher in hand than tool-selective parietal regions. Notably, grasp typicality representations were automatically evoked even when there was no requirement for tool use and participants were naive to object category (tool vs nontools). Finding a specificity for typical tool grasping in hand-selective, rather than tool-selective, regions challenges the long-standing assumption that activation for viewing tool images reflects sensorimotor processing linked to tool manipulation. Instead, our results show that typicality representations for tool grasping are automatically evoked in visual regions specialized for representing the human hand, the primary tool of the brain for interacting with the world.

Most neuroimaging experiments that investigate how tools and their actions are represented in the brain use visual paradigms where tools or hands are displayed as 2D images and no real movements are performed. These studies discovered selective visual responses in occipito-temporal and parietal cortices for viewing pictures of hands or tools, which are assumed to reflect action processing, but this has rarely been directly investigated. Here, we examined the responses of independently visually defined category-selective brain areas when participants grasped 3D tools. Using real action fMRI and multi-voxel pattern analysis, we found that grasp typicality representations (i.e., whether a tool is being grasped appropriately for use) were decodable from hand-selective areas in occipito-temporal and parietal cortices, but not from tool-, object-, or body-selective areas, even if partially overlapping. Importantly, these effects were exclusive for actions with tools, but not for biomechanically matched actions with control nontools. In addition, decoding of grasp typicality was significantly higher in hand than tool-selective parietal regions. Notably, grasp typicality representations were automatically evoked even when there was no requirement for tool use and participants were naïve to object category (tool vs non-tools). Finding a specificity for typical tool grasping in hand-, rather than tool-, selective regions challenges the long-standing assumption that brain activation for viewing tool images reflects sensorimotor processing linked to tool manipulation. Instead our results show that typicality representations for tool grasping are automatically evoked in visual regions specialised for representing the human hand, the brain’s primary tool for interacting with the world. Significance Statement: The unique ability of humans to manufacture and use tools is unsurpassed across the animal kingdom, with tool use considered a defining feature of our species. Most neuroscientific studies that investigate the brain mechanisms that support tool use, record brain activity while people simply view images of tools or hands and not when people perform actual hand movements with tools. Here we show that specific areas of the human visual system that preferentially process hands automatically encode how to appropriately grasp 3D tools, even when no actual tool use is required. These findings suggest that visual areas optimized for processing hands represent fundamental aspects of tool grasping in humans, such as which side they should be grasped for correct manipulation.

Neurons, even in earliest sensory regions of cortex, are subject to a great deal of contextual influences from both within and across modality connections. Recent work has shown that primary sensory areas can respond to and in some cases discriminate stimuli not of their target modality: for example, primary somatosensory cortex (SI) discriminates visual images of graspable objects. In the present work, we investigated whether SI would discriminate sounds depicting hand-object interactions (e.g. bouncing a ball). In a rapid event-related functional magnetic resonance imaging (fMRI) experiment, participants listened attentively to sounds from three categories: hand-object interactions, and control categories of pure tones and animal vocalizations, while performing a one-back repetition detection task. Multi-voxel pattern analysis revealed significant decoding of different hand-object interactions within SI, but not for either control category. Crucially, in the hand-sensitive voxels defined from an independent tactile localizer, decoding accuracies were significantly higher for hand-object interactions compared to pure tones in left SI. Our findings indicate that simply hearing sounds depicting familiar hand-object interactions elicit different patterns of activity in SI, despite the complete absence of tactile stimulation. These results highlight the rich information that can be transmitted across sensory modalities even to primary sensory areas.

Faces transmit a wealth of important social signals. While previous studies have elucidated the network of cortical regions important for perception of facial expression, and the associated temporal components such as the P100, N170 and EPN, it is still unclear how task constraints may shape the representation of facial expression (or other face categories) in these networks. In the present experiment, we investigate the neural information available across time about two important face categories (expression and identity) when those categories are either perceived under explicit (e.g. decoding emotion when task is on emotion) or implicit task contexts (e.g. decoding emotion when task is on identity). Decoding of both face categories, across both task contexts, peaked in a 100-200ms time-window post-stimulus (across posterior electrodes). Peak decoding of expression, however, was not affected by task context whereas peak decoding of identity was significantly reduced under implicit processing conditions. In addition, errors in EEG decoding correlated with errors in behavioral categorization under explicit processing for both expression and identity, but only with implicit decoding of expression. Despite these differences, decoding time-courses and the spatial pattern of informative electrodes differed consistently for both tasks across explicit Vs implicit face processing. Finally our results show that information about both face identity and facial expression is available around the N170 time-window on lateral occipito-temporal sites. Taken together, these results reveal differences and commonalities in the processing of face categories under explicit Vs implicit task contexts and suggest that facial expressions are processed to a richer degree even under implicit processing conditions, consistent with prior work indicating the relative automaticity by which emotion is processed. Our work further demonstrates the utility in applying multivariate decoding analyses to EEG for revealing the dynamics of face perception.