Zhe-Xin Xu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Zhe-Xin Xu has a total of 12 co-authors. Here are the top co-authors by number of shared publications:
- Gregory C. DeAngelis
Department of Brain and Cognitive Sciences, University of Rochester
3 shared publications - Shu-Guang Kuai
Shanghai Key Laboratory of Mental Health and Psychological Crisis Intervention and Key Laboratory of Brain Functional Genomics of Ministry of Education, School of Psychology and Cognitive Science, East China Normal University, Shanghai, China
3 shared publications Yan Chen
Shanghai Key Laboratory of Brain Functional Genomics, Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University
3 shared publicationsAkiyuki Anzai
Department of Brain and Cognitive Sciences, Center for Visual Science, University of Rochester, Rochester, USA
2 shared publicationsJia-Mei Li
Faculty of Education, Shanghai Key Laboratory of Brain Functional Genomics, School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China
2 shared publicationsJiayi Pang
Department of Brain and Cognitive Sciences, Center for Visual Science, University of Rochester, Rochester, USA
2 shared publications
Latest Publications
- Open Access
- 30 December 2025
- Published by Springer Nature in Nature Communications
- Vol. 17 (1)
Flexible computation of object motion and depth based on viewing geometry inferred from optic flow
We move our eyes and head to sample the visual environment. While these movements are essential for survival, they greatly complicate the analysis of retinal image motion. Our brain must account for the visual consequences of self-motion to perceive the 3D layout and motion of objects in a scene. We show that traditional models of visual compensation for eye movements fail when the eye both translates and rotates, and we propose a theory that computes both motion and depth in more natural viewing geometries. Consistent with our theoretical predictions, humans exhibit distinct perceptual biases when different viewing geometries are simulated by optic flow, and these biases occur without training or feedback. A neural network model trained to perform the same tasks suggests that viewing geometry modulates the joint tuning of neurons for retinal and eye velocity to mediate these adaptive computations. Our findings unify previously separate bodies of work by demonstrating that the brain adaptively perceives the dynamic 3D environment according to viewing geometry inferred from optic flow.
- 30 December 2025
- Published by Springer Nature in Nature Communications
- Vol. 17 (1)
Open AccessFlexible computation of object motion and depth based on viewing geometry inferred from optic flow
We move our eyes and head to sample the visual environment. While these movements are essential for survival, they greatly complicate the analysis of retinal image motion. Our brain must account for the visual consequences of self-motion to perceive the 3D layout and motion of objects in a scene. We show that traditional models of visual compensation for eye movements fail when the eye both translates and rotates, and we propose a theory that computes both motion and depth in more natural viewing geometries. Consistent with our theoretical predictions, humans exhibit distinct perceptual biases when different viewing geometries are simulated by optic flow, and these biases occur without training or feedback. A neural network model trained to perform the same tasks suggests that viewing geometry modulates the joint tuning of neurons for retinal and eye velocity to mediate these adaptive computations. Our findings unify previously separate bodies of work by demonstrating that the brain adaptively perceives the dynamic 3D environment according to viewing geometry inferred from optic flow.
- Open Access
- 13 October 2025
- Published by eLife Sciences Publications, Ltd in eLife preprints
Allocentric and egocentric cues constitute an internal reference frame for real-world visual search
Visual search in natural environments involves numerous objects, each composed of countless features. Despite this complexity, our brain efficiently locates targets. Here, we propose that the brain combines multiple reference cues to form an internal reference frame that facilitates real-world visual search. Objects in natural scenes often appear in orientations perceived as upright, enabling quicker recognition. However, how object orientation influences real-world visual search remains unknown. Moreover, the contributions of different reference cues—egocentric, visual context, and gravitational— are not well understood. To answer these questions, we designed a visual search task in virtual reality. Our results revealed an orientation effect independent of set size, suggesting reference frame transformation rather than object rotation. By rotating virtual scenes and participants in a flight simulator, we found that allocentric cues drastically altered search performance. These findings provide novel insights into the efficiency of real-world visual search and its connection to multimodal cognition.
- 13 October 2025
- Published by eLife Sciences Publications, Ltd in eLife preprints
Open AccessAllocentric and egocentric cues constitute an internal reference frame for real-world visual search
Visual search in natural environments involves numerous objects, each composed of countless features. Despite this complexity, our brain efficiently locates targets. Here, we propose that the brain combines multiple reference cues to form an internal reference frame that facilitates real-world visual search. Objects in natural scenes often appear in orientations perceived as upright, enabling quicker recognition. However, how object orientation influences real-world visual search remains unknown. Moreover, the contributions of different reference cues—egocentric, visual context, and gravitational— are not well understood. To answer these questions, we designed a visual search task in virtual reality. Our results revealed an orientation effect independent of set size, suggesting reference frame transformation rather than object rotation. By rotating virtual scenes and participants in a flight simulator, we found that allocentric cues drastically altered search performance. These findings provide novel insights into the efficiency of real-world visual search and its connection to multimodal cognition.
- Open Access
- 16 July 2025
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 45 (29)
A Normative Account of 3D Motion Perception from Natural Scenes
- 16 July 2025
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 45 (29)
Open AccessA Normative Account of 3D Motion Perception from Natural Scenes
- Open Access
Allocentric and egocentric cues constitute an internal reference frame for real-world visual search
Visual search in natural environments involves numerous objects, each composed of countless features. Despite this complexity, our brain efficiently locates targets. Here, we propose that the brain combines multiple reference cues to form an internal reference frame that facilitates real-world visual search. Objects in natural scenes often appear in orientations perceived as upright, enabling quicker recognition. However, how object orientation influences real-world visual search remains unknown. Moreover, the contributions of different reference cues—egocentric, visual context, and gravitational— are not well understood. To answer these questions, we designed a visual search task in virtual reality. Our results revealed an orientation effect independent of set size, suggesting reference frame transformation rather than object rotation. By rotating virtual scenes and participants in a flight simulator, we found that allocentric cues drastically altered search performance. These findings provide novel insights into the efficiency of real-world visual search and its connection to multimodal cognition. Significance: A central question in the behavioral sciences concerns how people efficiently perceive natural environments. Visual search exemplifies this challenge. While research has elucidated the basic mechanisms, traditional theories struggle to explain the remarkable efficiency in real-world scenes. Here, we examine a fundamental property of natural scenes: reference frames. Real-world objects typically appear in consistent orientations, suggesting that orientation may guide search. Yet, the influence of object orientation on real-world search—and which reference cues (egocentric, visual context, or gravitational) determine that orientation—remains unknown. We developed a novel virtual-reality paradigm to address these questions. We demonstrated that humans combine multiple reference cues to form an internal reference frame that guides visual search, providing a novel account of the efficiency of real-world search.
Open AccessAllocentric and egocentric cues constitute an internal reference frame for real-world visual search
Visual search in natural environments involves numerous objects, each composed of countless features. Despite this complexity, our brain efficiently locates targets. Here, we propose that the brain combines multiple reference cues to form an internal reference frame that facilitates real-world visual search. Objects in natural scenes often appear in orientations perceived as upright, enabling quicker recognition. However, how object orientation influences real-world visual search remains unknown. Moreover, the contributions of different reference cues—egocentric, visual context, and gravitational— are not well understood. To answer these questions, we designed a visual search task in virtual reality. Our results revealed an orientation effect independent of set size, suggesting reference frame transformation rather than object rotation. By rotating virtual scenes and participants in a flight simulator, we found that allocentric cues drastically altered search performance. These findings provide novel insights into the efficiency of real-world visual search and its connection to multimodal cognition. Significance: A central question in the behavioral sciences concerns how people efficiently perceive natural environments. Visual search exemplifies this challenge. While research has elucidated the basic mechanisms, traditional theories struggle to explain the remarkable efficiency in real-world scenes. Here, we examine a fundamental property of natural scenes: reference frames. Real-world objects typically appear in consistent orientations, suggesting that orientation may guide search. Yet, the influence of object orientation on real-world search—and which reference cues (egocentric, visual context, or gravitational) determine that orientation—remains unknown. We developed a novel virtual-reality paradigm to address these questions. We demonstrated that humans combine multiple reference cues to form an internal reference frame that guides visual search, providing a novel account of the efficiency of real-world search.
- Open Access
Flexible computation of object motion and depth based on viewing geometry inferred from optic flow
1 Vision is an active process. We move our eyes and head to acquire useful information and to track objects of interest. While these movements are essential for many behaviors, they greatly complicate the analysis of retinal image motion—the image motion of an object reflects both how that object moves in the world and how the eye moves relative to the scene. Our brain must account for the visual consequences of self-motion to accurately perceive the 3D layout and motion of objects in the scene. Traditionally, compensation for eye movements (e.g., smooth pursuit) has been modeled as a simple vector subtraction process. While these models are effective for pure eye rotations and 2D scenes, we show that they fail to apply to more natural viewing geometries involving combinations of eye rotation and translation. We develop theoretical predictions for how perception of object motion and depth should depend on the observer’s inferred viewing geometry. Through psychophysical experiments, we demonstrate novel perceptual biases that manifest when different viewing geometries are simulated by optic flow, in the absence of physical eye movements. Remarkably, these biases occur automatically, without training or feedback, and are well predicted by our theoretical framework. A neural network model trained to perform the same tasks exhibits neural response patterns similar to those observed in macaque area MT, suggesting a possible neural basis for these adaptive computations. Our findings demonstrate that the visual system automatically infers viewing geometry from optic flow and flexibly attributes components of image motion to either self-motion or depth structure according to the inferred geometry. Our findings unify previously separate bodies of work by showing that the visual consequences of self-motion play a crucial role in computing object motion and depth, thus enabling the visual system to adaptively perceive a dynamic 3D environment.
Open AccessFlexible computation of object motion and depth based on viewing geometry inferred from optic flow
1 Vision is an active process. We move our eyes and head to acquire useful information and to track objects of interest. While these movements are essential for many behaviors, they greatly complicate the analysis of retinal image motion—the image motion of an object reflects both how that object moves in the world and how the eye moves relative to the scene. Our brain must account for the visual consequences of self-motion to accurately perceive the 3D layout and motion of objects in the scene. Traditionally, compensation for eye movements (e.g., smooth pursuit) has been modeled as a simple vector subtraction process. While these models are effective for pure eye rotations and 2D scenes, we show that they fail to apply to more natural viewing geometries involving combinations of eye rotation and translation. We develop theoretical predictions for how perception of object motion and depth should depend on the observer’s inferred viewing geometry. Through psychophysical experiments, we demonstrate novel perceptual biases that manifest when different viewing geometries are simulated by optic flow, in the absence of physical eye movements. Remarkably, these biases occur automatically, without training or feedback, and are well predicted by our theoretical framework. A neural network model trained to perform the same tasks exhibits neural response patterns similar to those observed in macaque area MT, suggesting a possible neural basis for these adaptive computations. Our findings demonstrate that the visual system automatically infers viewing geometry from optic flow and flexibly attributes components of image motion to either self-motion or depth structure according to the inferred geometry. Our findings unify previously separate bodies of work by showing that the visual consequences of self-motion play a crucial role in computing object motion and depth, thus enabling the visual system to adaptively perceive a dynamic 3D environment.
- 19 February 2024
- Published by Springer Nature in Nature Reviews Psychology
- Vol. 3 (4) , 220
Probing the brain’s visual catalogue
- 19 February 2024
- Published by Springer Nature in Nature Reviews Psychology
- Vol. 3 (4) , 220
Probing the brain’s visual catalogue
- Open Access
Speed tuning in head coordinates as an alternative explanation of depth selectivity from motion parallax in area MT
There are two distinct sources of retinal image motion: motion of objects in the world and movement of the observer. In cases where an object moves in a scene and the eyes also move, a coordinate transformation that involves smooth eye movements and retinal motion will be needed in order to estimate object motion in world coordinates. More recently, interactions between retinal and eye velocity signals have also been suggested to generate depth selectivity from motion parallax (MP) in the macaque middle temporal (MT) area. We explored whether the nature of the interaction between eye and retinal velocities in MT neurons favors one of these two possibilities or a mixture of both. We analyzed responses of MT neurons to retinal and eye velocities in a viewing context in which the observer translates laterally while maintaining visual fixation on a world-fixed target. In this scenario, the depth of an object can be inferred from the ratio between retinal velocity and eye velocity, according to the motion-pursuit law. Previous studies have shown that MT responses to retinal motion are gain-modulated by the direction of eye movement, suggesting a potential mechanism for depth tuning from MP. However, our analysis of the joint tuning profile for retinal and eye velocities reveals that some MT neurons show a partial coordinate transformation toward head coordinates. We formalized a series of computational models to predict neural spike trains as well as selectivity for depth, and we used factorial model comparisons to quantify the relative importance of each model component. Our findings for many MT neurons reveal that the data are equally well explained by gain modulation or a partial coordinate transformation toward head coordinates, although some responses can only be well fit by the coordinate transform model. Our results highlight the potential role of MT neurons in representing multiple higher-level sensory variables, including depth from MP and object motion in the world.
Open AccessSpeed tuning in head coordinates as an alternative explanation of depth selectivity from motion parallax in area MT
There are two distinct sources of retinal image motion: motion of objects in the world and movement of the observer. In cases where an object moves in a scene and the eyes also move, a coordinate transformation that involves smooth eye movements and retinal motion will be needed in order to estimate object motion in world coordinates. More recently, interactions between retinal and eye velocity signals have also been suggested to generate depth selectivity from motion parallax (MP) in the macaque middle temporal (MT) area. We explored whether the nature of the interaction between eye and retinal velocities in MT neurons favors one of these two possibilities or a mixture of both. We analyzed responses of MT neurons to retinal and eye velocities in a viewing context in which the observer translates laterally while maintaining visual fixation on a world-fixed target. In this scenario, the depth of an object can be inferred from the ratio between retinal velocity and eye velocity, according to the motion-pursuit law. Previous studies have shown that MT responses to retinal motion are gain-modulated by the direction of eye movement, suggesting a potential mechanism for depth tuning from MP. However, our analysis of the joint tuning profile for retinal and eye velocities reveals that some MT neurons show a partial coordinate transformation toward head coordinates. We formalized a series of computational models to predict neural spike trains as well as selectivity for depth, and we used factorial model comparisons to quantify the relative importance of each model component. Our findings for many MT neurons reveal that the data are equally well explained by gain modulation or a partial coordinate transformation toward head coordinates, although some responses can only be well fit by the coordinate transform model. Our results highlight the potential role of MT neurons in representing multiple higher-level sensory variables, including depth from MP and object motion in the world.
- Open Access
- 11 December 2019
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 40 (5) , 1120-1132
Integration of Motion and Form Cues for the Perception of Self-Motion in the Human Brain
When moving around in the world, the human visual system uses both motion and form information to estimate the direction of self-motion (i.e., heading). However, little is known about cortical areas in charge of this task. This brain-imaging study addressed this question by using visual stimuli consisting of randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion [FoE]) but moved away from a different locus (the motion-defined FoE) to simulate observer translation. We first fixed the motion-defined FoE location and shifted the form-defined FoE location. We then made the locations of the motion- and the form-defined FoEs either congruent (at the same location in the display) or incongruent (on the opposite sides of the display). The motion- or the form-defined FoE shift was the same in the two types of stimuli, but the perceived heading direction shifted for the congruent, but not for the incongruent stimuli. Participants (both sexes) made a task-irrelevant (contrast discrimination) judgment during scanning. Searchlight and ROI-based multivoxel pattern analysis revealed that early visual areas V1, V2, and V3 responded to either the motion- or the form-defined FoE shift. After V3, only the dorsal areas V3a and V3B/KO responded to such shifts. Furthermore, area V3B/KO shows a significantly higher decoding accuracy for the congruent than the incongruent stimuli. Our results provide direct evidence showing that area V3B/KO does not simply respond to motion and form cues but integrates these two cues for the perception of heading.SIGNIFICANCE STATEMENTHuman survival relies on accurate perception of self-motion. The visual system uses both motion (optic flow) and form cues for the perception of the direction of self-motion (heading). Although human brain areas for processing optic flow and form structure are well identified, the areas responsible for integrating these two cues for the perception of self-motion remain unknown. We conducted fMRI experiments and used multivoxel pattern analysis technique to find human brain areas that can decode the shift in heading specified by each cue alone and the two cues combined. We found that motion and form cues are first processed in the early visual areas and then are likely integrated in the higher dorsal area V3B/KO for the final estimation of heading.
- 11 December 2019
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 40 (5) , 1120-1132
Open AccessIntegration of Motion and Form Cues for the Perception of Self-Motion in the Human Brain
When moving around in the world, the human visual system uses both motion and form information to estimate the direction of self-motion (i.e., heading). However, little is known about cortical areas in charge of this task. This brain-imaging study addressed this question by using visual stimuli consisting of randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion [FoE]) but moved away from a different locus (the motion-defined FoE) to simulate observer translation. We first fixed the motion-defined FoE location and shifted the form-defined FoE location. We then made the locations of the motion- and the form-defined FoEs either congruent (at the same location in the display) or incongruent (on the opposite sides of the display). The motion- or the form-defined FoE shift was the same in the two types of stimuli, but the perceived heading direction shifted for the congruent, but not for the incongruent stimuli. Participants (both sexes) made a task-irrelevant (contrast discrimination) judgment during scanning. Searchlight and ROI-based multivoxel pattern analysis revealed that early visual areas V1, V2, and V3 responded to either the motion- or the form-defined FoE shift. After V3, only the dorsal areas V3a and V3B/KO responded to such shifts. Furthermore, area V3B/KO shows a significantly higher decoding accuracy for the congruent than the incongruent stimuli. Our results provide direct evidence showing that area V3B/KO does not simply respond to motion and form cues but integrates these two cues for the perception of heading.SIGNIFICANCE STATEMENTHuman survival relies on accurate perception of self-motion. The visual system uses both motion (optic flow) and form cues for the perception of the direction of self-motion (heading). Although human brain areas for processing optic flow and form structure are well identified, the areas responsible for integrating these two cues for the perception of self-motion remain unknown. We conducted fMRI experiments and used multivoxel pattern analysis technique to find human brain areas that can decode the shift in heading specified by each cue alone and the two cues combined. We found that motion and form cues are first processed in the early visual areas and then are likely integrated in the higher dorsal area V3B/KO for the final estimation of heading.
- Open Access
- 3 December 2018
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 18 (13) , 10
The human visual system estimates angle features in an internal reference frame: A computational and psychophysical study
Angle perception is an important middle-level visual process, combining line features to generate an integrated shape percept. Previous studies have proposed two theories of angle perception—a combination of lines and a holistic feature following Weber's law. However, both theories failed to explain the dual-peak fluctuations of the just-noticeable difference (JND) across angle sizes. In this study, we found that the human visual system processes the angle feature in two stages: first, by encoding the orientation of the bounding lines and combining them into an angle feature; and second, by estimating the angle in an orthogonal internal reference frame (IRF). The IRF model fits well with the dual-peak fluctuations of the JND that neither the theory of line combinations nor Weber's law can explain. A statistical image analysis of natural images revealed that the IRF was in alignment with the distribution of the angle features in the natural environment, suggesting that the IRF reflects human prior knowledge of angles in the real world. This study provides a new computational framework for angle discrimination, thereby resolving a long-standing debate on angle perception.
- 3 December 2018
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 18 (13) , 10
Open AccessThe human visual system estimates angle features in an internal reference frame: A computational and psychophysical study
Angle perception is an important middle-level visual process, combining line features to generate an integrated shape percept. Previous studies have proposed two theories of angle perception—a combination of lines and a holistic feature following Weber's law. However, both theories failed to explain the dual-peak fluctuations of the just-noticeable difference (JND) across angle sizes. In this study, we found that the human visual system processes the angle feature in two stages: first, by encoding the orientation of the bounding lines and combining them into an angle feature; and second, by estimating the angle in an orthogonal internal reference frame (IRF). The IRF model fits well with the dual-peak fluctuations of the JND that neither the theory of line combinations nor Weber's law can explain. A statistical image analysis of natural images revealed that the IRF was in alignment with the distribution of the angle features in the natural environment, suggesting that the IRF reflects human prior knowledge of angles in the real world. This study provides a new computational framework for angle discrimination, thereby resolving a long-standing debate on angle perception.
- Open Access
- 1 September 2018
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 18 (10) , 44
Cortical areas that integrate motion and form cues for the perception of self-motion
When moving around in the world, the human visual system uses both form and motion information to estimate the direction of self-motion (i.e., heading). Here we explored brain areas that integrate motion and form cues for heading perception. We used stimuli consisting of 200 randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion (FOE)) but moved away from a different locus (the motion-defined FOE) to simulate observer translation. In Experiment 1, we fixed the motion-defined FOE at the display center and shifted the form-defined FOE from -5° to 5° at the step of 2°. In Experiment 2, the form- and motion-defined FOEs were congruent (i.e., at the same location in the display) or incongruent (i.e., on the opposite side of the display) but had the same shifts. Participants made a task-irrelevant (luminance discrimination) judgment during scanning. We preformed the searchlight and ROI-based multiple voxel pattern analysis and found that early visual areas (V1, V2) decoded both form- and motion-defined FOEs but could not discriminate the congruent and incongruent conditions, suggesting that these areas do not integrate form and motion cues for heading perception. The higher ventral areas (V3v, V4v) decoded form- but not motion-defined FOE while the dorsal areas (MT+) and parietal lobe (VIP , CSV) decoded motion- but not form-defined FOE. In contrast, the higher dorsal areas (V3d, V3a, V7, KO) not only decoded both form- and motion-defined FOEs but also dissociated the congruent and incongruent conditions, suggesting that they contribute to the integration of form and motion cues for heading perception. Our findings provide the first empirical evidence suggesting that form and motion information are first processed along separate pathways and then integrated in the higher dorsal areas for the final estimation of heading during self-motion. Meeting abstract presented at VSS 2018
- 1 September 2018
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 18 (10) , 44
Open AccessCortical areas that integrate motion and form cues for the perception of self-motion
When moving around in the world, the human visual system uses both form and motion information to estimate the direction of self-motion (i.e., heading). Here we explored brain areas that integrate motion and form cues for heading perception. We used stimuli consisting of 200 randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion (FOE)) but moved away from a different locus (the motion-defined FOE) to simulate observer translation. In Experiment 1, we fixed the motion-defined FOE at the display center and shifted the form-defined FOE from -5° to 5° at the step of 2°. In Experiment 2, the form- and motion-defined FOEs were congruent (i.e., at the same location in the display) or incongruent (i.e., on the opposite side of the display) but had the same shifts. Participants made a task-irrelevant (luminance discrimination) judgment during scanning. We preformed the searchlight and ROI-based multiple voxel pattern analysis and found that early visual areas (V1, V2) decoded both form- and motion-defined FOEs but could not discriminate the congruent and incongruent conditions, suggesting that these areas do not integrate form and motion cues for heading perception. The higher ventral areas (V3v, V4v) decoded form- but not motion-defined FOE while the dorsal areas (MT+) and parietal lobe (VIP , CSV) decoded motion- but not form-defined FOE. In contrast, the higher dorsal areas (V3d, V3a, V7, KO) not only decoded both form- and motion-defined FOEs but also dissociated the congruent and incongruent conditions, suggesting that they contribute to the integration of form and motion cues for heading perception. Our findings provide the first empirical evidence suggesting that form and motion information are first processed along separate pathways and then integrated in the higher dorsal areas for the final estimation of heading during self-motion. Meeting abstract presented at VSS 2018