Filipp Schmidt

How do we make sense of something we’ve never seen before? Classifying objects into superordinate classes like ‘animal’ is a key step in interpreting novel experiences, but is challenging because radically different items (e.g., octopus, rabbit) must somehow be grouped together. In general, no single feature is shared by all members. We reasoned that to classify or imagine novel items from outside the distribution of previous experiences, observers parse objects into meaningful component features that they can mentally recombine (‘compositionality’). To test this, we asked participants to draw familiar and novel members of nine superordinate object classes. We then asked other participants to classify the drawings, and mark and label their defining ‘parts’. We find that human classification performance is well predicted by a Bayesian classifier that optimally combines the part labels, suggesting humans can create and classify out-of-distribution experiences through a compositional generative representation of object features.

Materials from distinct categories share common image features but also exhibit unique characteristics. How do humans recognize materials despite their enormous appearance variations? Using generative neural networks, we create both prototypical and ambiguous materials that morph between categories. To capture the richness of material representation, we characterize how people judge materials using cross-material morphs through three behavioral tasks: material categorization, material property rating, and visual similarity judgment. We find that morphing smoothly modulates perceptual scales of material appearance. Despite this smooth variation, participants can reliably identify the prototypical materials within given appearance ranges, indicating a strong association between material categories and their visual characteristics. Material properties structure the perceptual space, where the salient dimensions--particularly rigidity--strongly correlate with perceived material categories. In contrast, analyzing image embedding derived from a self-supervised deep learning model reveals that learning visual similarities alone is insufficient to reproduce human perceptual space of materials. Together, our results suggest that material representation is more than visual similarities and may require learning material properties that structure relationships among materials within and across categories. Visual features might be used flexibly and task-dependently to support multiple levels of material perception, ranging from discrimination to material property inference to categorization.

Detecting whether others are approaching or moving away is fundamental for adaptive behavior. This is particularly true under uncertainty where perceptual interpretations are shaped by expectations and affective states such as fear. The present study examined whether anxiety-relevant stimuli systematically bias directional perception when biological motion is ambiguous. We used point-light walkers—minimal displays of moving dots marking the major joints of an agent—to investigate biological motion perception. When orthographically projected onto the frontal plane, these displays are perceptually bistable and can be interpreted as either approaching or receding. Observers typically resolve this ambiguity by perceiving the agent as approaching, a phenomenon known as the facing-the-viewer bias. Despite its potential significance for threat detection, evidence regarding the influence of anxiety-related traits on the bias has been inconsistent. We therefore measured the facing-the-viewer bias across point-light walkers of different species (human, cat, spider) and walking orientations to study its modulation by spider fear. For this, participants with varying levels of spider fear (N = 100) completed a perceptual judgment task indicating the perceived walking orientation of point-light walkers. Results revealed a robust facing-the-viewer bias across species and orientations. Critically, higher spider fear was associated with a stronger bias particularly for frontally oriented spiders. These findings suggest that the facing-the-viewer bias reflects a core sociobiological mechanism for prioritizing potentially approaching agents, which becomes selectively amplified in the presence of fear-relevant stimuli—underscoring how affective expectations can shape perceptual interpretation during ambiguous encounters with feared animals.

In amodal completion observers perceive complete objects despite partial occlusion. When two object parts are divided by an occluder, completion can result in perceiving one or two objects. This phenomenon involves both lower-level cues (e.g., symmetry, contour continuity) and higher-level cues (e.g., prior knowledge). Experiment 1 investigates how occluder size, familiarity, and symmetry affect human completions using a drawing task. Narrow occluders and asymmetry promote single-shape completions, while familiarity and (global) symmetry promote two-shape interpretations. Good continuation emerges as the strongest cue, with symmetry and familiarity playing increasingly important roles as occluder width increases. Experiment 2 compares human performance with three state-of-the-art generative AI models. Models often generated creative but non-compliant outputs, altering even unoccluded regions. We restricted analysis to instruction-following generations, identified through ratings by naive observers. Among compliant outputs, models showed some human-like biases (e.g., more two-shape completions for wide occluders), but failed with higher-level cues. They did not use symmetry to guide completions and showed reversed familiarity effects. Our findings highlight differences between human and AI completions. Humans integrate low- and high-level cues, whereas compliant outputs from the AI models rely primarily on low-level pattern continuation. Current AI models lack the flexible integration of multiple representational levels that characterize human perception. This work establishes an analytical framework for evaluating whether next-generation models achieve more human-like visual reasoning. Highlights: Humans see one or two objects behind occluders using geometric and semantic cues. Human drawings and generative AI completions show how different cues modulate perception Good continuation dominates human and AI shape completions Symmetry and familiarity guide humans, but not instruction-following AI outputs Humans integrate multiple levels; instruction-following AI engages lower-level-processing.

Physiological and psychophysical evidence suggests that the visual system represents object outlines using prominent curvature features, particularly regions of extreme curvature (convex maxima and concave minima). These curvature extrema often coincide with points of high informational content (“surprisal”), but this relationship is only correlational. It remains unclear whether the visual system explicitly encodes curvature extrema or instead prioritizes the most informative contour locations. To address this, we conducted two shape-matching experiments comparing the roles of curvature extrema and surprisal in shape representation. Observers performed match-to-sample tasks in which smooth reference shapes were matched to simplified polygonal versions created by connecting subsets of contour points corresponding to (i) curvature maxima, (ii) curvature maxima and minima, or (iii) points of highest surprisal. Stimuli included artificial shapes composed of compound radial frequency patterns and natural shapes (animal outlines), the latter allowing us to dissociate curvature and information by restricting sampled points. Performance was higher for natural than artificial shapes (95% vs. ∼86%). Shapes defined by a small number of high-surprisal points matched performance in baseline and curvature maxima and minima conditions, but exceeded performance for curvature maxima alone (∼90% vs. ∼65%). In a second experiment, we contrasted surprisal with curvature maxima and minima conditions while varying the number of sampled points (4–32). Performance increased with point number, approaching ∼90%. Critically, under strong simplification (4–6 points), surprisal-based shapes yielded higher accuracy than curvature-based shapes. These findings suggest that shape representation emphasizes features with high informational content rather than curvature extrema per se.

Digital representation of materials is crucial in fields such as virtual reality, industrial design and quality control. However, predicting human perception of materials from image data is challenging due to the complexity of material appearances and the intricacies of human vision. This study introduces a perceptual representation termed the ‘visual fingerprint’, linking image-based measurements of materials to intuitive, human-understandable attributes. We conducted psychophysical studies using standardized video sequences of 347 diverse real-world materials, including fabrics and wood, selected to encompass a broad spectrum of textures, colours and reflective properties. Sixteen key appearance attributes were identified, and over 110 000 human ratings were collected to map perceptual attributes across material categories. By integrating CLIP-derived image features with a multi-layer perceptron model, we developed a predictive framework for material perception. Our results demonstrate that human judgements of appearance and similarity can be accurately predicted using only two images of a material. This work offers a practical and interpretable approach to material representation, enabling intuitive comparisons and retrievals in applications where material appearance is crucial. The proposed material fingerprint and its prediction directly from image data represent a significant step towards simplifying the understanding and interoperability of material properties in diverse digital environments.

Visually categorizing and comparing materials is crucial for everyday behavior, but what organizational principles underlie our mental representation of materials? Here, we used a large-scale data-driven approach to uncover core latent dimensions of material representations from behavior. First, we created an image dataset of 200 systematically sampled materials and 600 photographs (STUFF dataset, https://osf.io/myutc/ ). Using these images, we next collected 1.87 million triplet similarity judgments and used a computational model to derive a set of sparse, positive dimensions underlying these judgments. The resulting multidimensional embedding space predicted independent material similarity judgments and the similarity matrix of all images close to the human intersubject consistency. We found that representations of individual images were captured by a combination of 36 material dimensions that were highly reproducible and interpretable, comprising perceptual (e.g., grainy, blue) as well as conceptual (e.g., mineral, viscous) dimensions. These results provide the foundation for a comprehensive understanding of how humans make sense of materials.