JAJan Jaap R. van Assen

Jan Jaap R. van Assen

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Jan Jaap R. van Assen has a total of 12 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Material fictions: Comparing physically based renderings and generative AI images through material perception

    by Yuguang Zhao, Jeroen Stumpel, Huib de Ridder, Jan Jaap R. van Assen, Maarten W. A. Wijntjes

    Generative artificial intelligence (AI) models unlock new ways to create images, emerging as a new medium alongside paintings, photographs, physically based renderings (PBR), etc. Generative AI images can be perceptually convincing without being physically plausible, allowing to investigate the boundaries of visual perception. This study examines whether generative AI images adhere to a medium-independent perceptual space converged from previous studies. We compared the perceptual similarity of images from three generative AI models against a bidirectional reflectance distribution functions (BRDFs) PBR image dataset, using human similarity judgments. In experiment 1, we used the text descriptions of 32 materials (e.g., blue acrylic) from the Mitsubishi Electric Research Laboratories (MERL) BRDF dataset, prompting two text-to-image models, DALL-E 2 and Midjourney v2, to generate 32 sphere-shaped stimuli per model. Perceptual spaces derived from similarity judgments revealed that both AI models resulted in two-dimensional spaces whereas the MERL space was confined to one dimension, probably owing to a lack of surface texture. These unrelated perceptual spaces suggest the AI models generated unique and different images from identical text prompts. In experiment 2 we used the text-to-image model Stable Diffusion v1.5 with ControlNet for additional depth-map constraints. Using the same 32 descriptions, we generated 3 sets using 3 different depth maps. The three resulting perceptual spaces are all two-dimensional, exhibiting high similarity, indicating a robust and non-random structure. They also show a similar structure to the MERL space and perceptual spaces from other material studies using photographs, PBR, and depictions, suggesting AI-generated imagery may indeed be used as a new medium to explore material perception.


  • Open Access

    Visual perception of liquids: Insights from deep neural networks

    by Jan Jaap R. van Assen, Shin’yA Nishida, Roland W. Fleming

    Visually inferring material properties is crucial for many tasks, yet poses significant computational challenges for biological vision. Liquids and gels are particularly challenging due to their extreme variability and complex behaviour. We reasoned that measuring and modelling viscosity perception is a useful case study for identifying general principles of complex visual inferences. In recent years, artificial Deep Neural Networks (DNNs) have yielded breakthroughs in challenging real-world vision tasks. However, to model human vision, the emphasis lies not on best possible performance, but on mimicking the specific pattern of successes and errors humans make. We trained a DNN to estimate the viscosity of liquids using 100.000 simulations depicting liquids with sixteen different viscosities interacting in ten different scenes (stirring, pouring, splashing, etc). We find that a shallow feedforward network trained for only 30 epochs predicts mean observer performance better than most individual observers. This is the first successful image-computable model of human viscosity perception. Further training improved accuracy, but predicted human perception less well. We analysed the network’s features using representational similarity analysis (RSA) and a range of image descriptors (e.g. optic flow, colour saturation, GIST). This revealed clusters of units sensitive to specific classes of feature. We also find a distinct population of units that are poorly explained by hand-engineered features, but which are particularly important both for physical viscosity estimation, and for the specific pattern of human responses. The final layers represent many distinct stimulus characteristics—not just viscosity, which the network was trained on. Retraining the fully-connected layer with a reduced number of units achieves practically identical performance, but results in representations focused on viscosity, suggesting that network capacity is a crucial parameter determining whether artificial or biological neural networks use distributed vs. localized representations. How the brain visually computes the physical properties of complex natural materials is a major open challenge in visual neuroscience. Here, we focussed on the perception of liquids—a particularly challenging class of materials due to their extreme mutability and diverse behaviours. We present the first image-computable model that can predict average human viscosity judgments from fluid simulation movies as well as individual observers can across a wide range of viewing conditions. We trained artificial neural networks to estimate viscosity from 100,000 20-frame simulations, and find that the models best predict human perception after relatively little training—long before they have reached optimal performance. This suggests that while human viscosity perception is remarkably good, even better performance is theoretically possible. Probing the networks with ‘virtual electrophysiology’ reveals many different features the networks use to estimate viscosity. Surprisingly, we find that the represented features are highly influenced by the size of the networks’ parameter space, while prediction performance remains practically unchanged. This implies that some caution is required in making direct inferences between neural network models and the human visual system. However, the methods presented here provide a systematic framework for comparing humans to neural networks.


  • Open Access

    Visual perception of liquids: insights from deep neural networks

    by Jan Jaap R Van Assen, Shin’yA Nishida, Roland W Fleming

    Perceiving viscosity is challenging because liquids exhibit incredibly diverse behaviors across scenes. To gain insights into how the visual system estimates viscosity, we developed an artificial network consisting of three convolution layers and one fully-connected layer. We trained it to estimate viscosity in 100,000 computer-generated animations of liquids with 16 different viscosities across 10 scenes with diverse liquid interactions, e.g. stirring or gushing. As expected, with sufficient training, the network predicts physical viscosity nearly perfectly for the training set. Importantly, in the initial training phase, it showed a pattern of errors more similar to human observers for both trained and untrained sets. We analyzed the representations learned by the network using representational similarity analysis. Surprisingly, viscosity (either physical or perceptual) had little explanatory power on the activation pattern of the 4096 units in the final fully-connected (FC) layer. We suspected that the final layer has a rich representational capacity so that it could represent seemingly task-irrelevant stimulus dimensions in addition to the task-relevant one. Indeed, we found it easy to read out scene category from the FC activation patterns. When we reduced the number of units in the FC layer to 40 or less (which had little effect on viscosity prediction performance), viscosity explained the FC activation patterns very well. Inspired by classical psychophysical and neurophysiology experiments, we then probed artificial neural responses with low-level stimuli isolating specific stimulus characteristics. This allowed us to identify nodes sensitive to specific orientations, spatial frequencies, motion types, etc., and to test the extent to which high-level responses can be understood as their weighted combinations. Together, these observations suggest that the nature of the representation depends on the capacity of the network, and this is likely the case for biological networks as well.

  • Open Access

    Visual Features in the Perception of Liquids

    by Jan Jaap R. van Assen, Pascal Barla, Roland W. Fleming


  • Open Access

    Inferring the stiffness of unfamiliar objects from optical, shape, and motion cues

    by Filipp Schmidt, Vivian C. Paulun, Jan Jaap R. van Assen, Roland W. Fleming

    Visually inferring the stiffness of objects is important for many tasks but is challenging because, unlike optical properties (e.g., gloss), mechanical properties do not directly affect image values. Stiffness must be inferred either (a) by recognizing materials and recalling their properties (associative approach) or (b) from shape and motion cues when the material is deformed (estimation approach). Here, we investigated interactions between these two inference types. Participants viewed renderings of unfamiliar shapes with 28 materials (e.g., nickel, wax, cork). In Experiment 1, they viewed nondeformed, static versions of the objects and rated 11 material attributes (e.g., soft, fragile, heavy). The results confirm that the optical materials elicited a wide range of apparent properties. In Experiment 2, using a blue plastic material with intermediate apparent softness, the objects were subjected to physical simulations of 12 shape-transforming processes (e.g., twisting, crushing, stretching). Participants rated softness and extent of deformation. Both correlated with the physical magnitude of deformation. Experiment 3 combined variations in optical cues with shape cues. We find that optical cues completely dominate. Experiment 4 included the entire motion sequence of the deformation, yielding significant contributions of optical as well as motion cues. Our findings suggest participants integrate shape, motion, and optical cues to infer stiffness, with optical cues playing a major role for our range of stimuli.


  • Open Access

    Shape, motion, and optical cues to stiffness of elastic objects

    by Vivian C. Paulun, Filipp Schmidt, Jan Jaap R. van Assen, Roland W. Fleming

    Nonrigid materials, such as jelly, rubber, or sponge move and deform in distinctive ways depending on their stiffness. Which cues do we use to infer stiffness? We simulated cubes of varying stiffness and optical appearance (e.g., wood, metal, wax, jelly) being subjected to two kinds of deformation: (a) a rigid cylinder pushing downwards into the cube to various extents (shape change, but little motion: shape dominant), (b) a rigid cylinder retracting rapidly from the cube (same initial shapes, differences in motion: motion dominant). Observers rated the apparent softness/hardness of the cubes. In the shape-dominant condition, ratings mainly depended on how deeply the rod penetrated the cube and were almost unaffected by the cube's intrinsic physical properties. In contrast, in the motion-dominant condition, ratings varied systematically with the cube's intrinsic stiffness, and were less influenced by the extent of the perturbation. We find that both results are well predicted by the absolute magnitude of deformation, suggesting that when asked to judge stiffness, observers resort to simple heuristics based on the amount of deformation. Softness ratings for static, unperturbed cubes varied substantially and systematically depending on the optical properties. However, when animated, the ratings were again dominated by the extent of the deformation, and the effect of optical appearance was negligible. Together, our results suggest that to estimate stiffness, the visual system strongly relies on measures of the extent to which an object changes shape in response to forces.


  • Open Access

    Influence of optical material properties on the perception of liquids

    by Jan Jaap R. van Assen, Roland W. Fleming

    In everyday life we encounter a wide range of liquids (e.g., water, custard, toothpaste) with distinctive optical appearances and viscosities. Optical properties (e.g., color, translucency) are physically independent of viscosity, but, based on experience with real liquids, we may associate specific appearances (e.g., water, caramel) with certain viscosities. Conversely, the visual system may discount optical properties, enabling “viscosity constancy” based primarily on the liquid's shape and motion. We investigated whether optical characteristics affect the perception of viscosity and other properties of liquids. We simulated pouring liquids with viscosities ranging from water to molten glass and rendered them with nine different optical characteristics. In Experiment 1, observers (a) adjusted a match stimulus until it had the same perceived viscosity as a test stimulus with different optical properties, and (b) rated six physical properties of the test stimuli (runniness, shininess, sliminess, stickiness, warmth, wetness). We tested moving and static stimuli. In Experiment 2, observers had to associate names with every liquid in the stimulus set. We find that observers' viscosity matches correlated strongly with the true viscosities and that optical properties had almost no effect. However, some ratings of liquid properties did show substantial interactions between viscosity and optical properties. Observers associate liquid names primarily with optical cues, although some materials are associated with a specific viscosity or combination of viscosity and optics. These results suggest viscosity is inferred primarily from shape and motion cues but that optical characteristics influence recognition of specific liquids and inference of other physical properties.


  • Open Access

    Highlight shapes and perception of gloss for real and photographed objects

    by Jan Jaap R. van Assen, Maarten W. A. Wijntjes, Sylvia C. Pont

    Gloss perception strongly depends on the three-dimensional shape and the illumination of the object under consideration. In this study we investigated the influence of the spatial structure of the illumination on gloss perception. A diffuse light box in combination with differently shaped masks was used to produce a set of six simple and complex highlight shapes. The geometry of the simple highlight shapes was inspired by conventional artistic practice (e.g., ring flash for photography, window shape for painting and disk or square for cartoons). In the box we placed spherical stimuli that were painted in six degrees of glossiness. This resulted in a stimulus set of six highlight shapes and six gloss levels, a total of 36 stimuli. We performed three experiments of which two took place using digital photographs on a computer monitor and one with the real spheres in the light box. The observers had to perform a comparison task in which they chose which of two stimuli was glossiest and a rating task in which they rated the glossiness. The results show that, perhaps surprisingly, more complex highlight shapes were perceived to produce a less glossy appearance than simple highlight shapes such as a disk or square. These findings were confirmed for both viewing conditions, on a computer display and in a real setting. The results show that variations in the spatial structure of “rather simple” illumination of the “extended source” type highlight influences perceived glossiness.