MZMitchell J. P. van Zuijlen

Mitchell J. P. van Zuijlen

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Mitchell J. P. van Zuijlen has a total of 12 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Time of day perception in paintings

    by Cehao Yu, Mitchell J.P. van Zuijlen, Cristina Spoiala, Sylvia C. Pont, Maarten Wijntjes, Anya Hurlbert

    The spectral shape, irradiance, direction, and diffuseness of daylight vary regularly throughout the day. The variations in illumination and their effect on the light reflected from objects may in turn provide visual information as to the time of day. We suggest that artists’ color choices for paintings of outdoor scenes might convey this information and that therefore the time of day might be decoded from the colors of paintings. Here we investigate whether human viewers’ estimates of depicted time of day in paintings correlate with their image statistics, specifically chromatic and luminance variations. We tested time-of-day perception in 17th-20th century Western European paintings via two online rating experiments. In Experiment 1, viewers’ ratings from 7 time choices varied significantly and largely consistently across paintings, but with some ambiguity between morning and evening depictions. Analysis of the relationship between image statistics and ratings revealed correlations with the perceived time of day: higher “morningness” ratings associated with higher brightness, contrast, saturation, and darker yellow/ brighter blue hues; “eveningness” with lower brightness, contrast, saturation, and darker blue/ brighter yellow hues. Multiple linear regressions of extracted principal components yielded a predictive model that explained 76% of the variance in time-of-day perception. In Experiment 2, viewers rated paintings as morning or evening only; rating distributions differed significantly across paintings; and image statistics predicted people’s perceptions. These results suggest that artists used different color palettes and patterns to depict different times of day, and the human visual system holds consistent assumptions about the variation of natural light depicted in paintings.

  • Open Access

    Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision

    by Mitchell J. P. Van Zuijlen, Hubert Lin, Kavita Bala, Sylvia C. Pont, Maarten W. A. Wijntjes

    In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading atmaterialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.


  • Open Access

    What Can Style Transfer and Paintings Do For Model Robustness?

    by Hubert Lin, Mitchell van Zuijlen, Sylvia C. Pont, Maarten W. A. Wijntjes, Kavita Bala

    A common strategy for improving model robustness is through data augmentations. Data augmentations encourage models to learn desired invariances, such as invariance to horizontal flipping or small changes in color. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation to encourage invariance to textures by creating painting-like images from photographs. However, a stylized photograph is not quite the same as an artist-created painting. Artists depict perceptually meaningful cues in paintings so that humans can recognize salient components in scenes, an emphasis which is not enforced in style transfer. Therefore, we study how style transfer and paintings differ in their impact on model robustness. First, we investigate the role of paintings as style images for stylization-based data augmentation. We find that style transfer functions well even without paintings as style images. Second, we show that learning from paintings as a form of perceptual data augmentation can improve model robustness. Finally, we investigate the invariances learned from stylization and from paintings, and show that models learn different invariances from these differing forms of data. Our results provide insights into how stylization improves model robustness, and provide evidence that artist-created paintings can be a valuable source of data for model robustness.

  • Open Access

    Soft like velvet and shiny like satin: Perceptual material signatures of fabrics depicted in 17th century paintings

    by Francesca Di Cicco, Mitchell J. P. van Zuijlen, Maarten W. A. Wijntjes, Sylvia C. Pont

    Dutch 17th century painters were masters in depicting materials and their properties in a convincing way. Here, we studied the perception of the material signatures and key image features of different depicted fabrics, like satin and velvet. We also tested whether the perception of fabrics depicted in paintings related to local or global cues, by cropping the stimuli. In Experiment 1, roughness, warmth, softness, heaviness, hairiness, and shininess were rated for the stimuli shown either full figure or cropped. In the full figure, all attributes except shininess were rated higher for velvet, whereas shininess was rated higher for satin. This distinction was less clear in the cropped condition, and some properties were perceived significantly different between the two conditions. In Experiment 2 we tested whether this difference was due to the choice of the cropped area. On the basis of the results of Experiment 1, shininess and softness were rated for multiple crops from each fabric. Most crops from the same fabric differed significantly in shininess, but not in softness perception. Perceived shininess correlated positively with the mean luminance of the crops and the highlights’ coverage. Experiment 1 showed that painted velvet and satin triggered distinct perceptions, indicative of robust material signatures of the two fabrics. The results of Experiment 2 suggest that the presence of local image cues affects the perception of optical properties like shininess, but not mechanical properties such as softness.


  • Open Access

    Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision

    by Mitchell J. P. van Zuijlen, Hubert Lin, Kavita Bala, Sylvia C. Pont, Maarten W. A. Wijntjes

    A painter is free to modify how components of a natural scene are depicted, which can lead to a perceptually convincing image of the distal world. This signals a major difference between photos and paintings: paintings are explicitly created for human perception. Studying these painterly depictions could be beneficial to a multidisciplinary audience. In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists' eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse label (e.g., fabric) and a fine-grained label (e.g., velvety, silky). We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across art history, human perception, and computer vision. Our experiments include analyzing the distribution of materials depicted in paintings, showing how painters create convincing depictions using a stylized approach, and demonstrating how paintings can be used to build more robust computer vision models. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines. The MIP dataset is freely accessible at https://materialsinpaintings.tudelft.nl

  • Open Access

    A Database of Painterly Material Depictions

    by Mitchell Van Zuijlen, Hubert Lin, Kavita Bala, Sylvia C. Pont, Maarten W.A. Wijntjes

    Painters depict materials by utilizing an implicit knowledge of human material perception. To enable studies of this implicit knowledge, we created a database of paintings annotated with bounding boxes of various depicted materials (e.g., fabric, stone, wood, etc.). First, we collected a set of nearly 20K paintings (primarily Old Masters) from online open-access galleries of nine internationally renowned art institutions. For 14 material categories, we asked human annotators from Amazon Mechanical Turk (AMT) to indicate the presence of each material within each painting. On average, we find 6.4 unique materials per painting. Skin and fabric were the most frequently depicted materials, while ceramic and food were the least frequent. Additionally, we calculated material co-occurrence correlations which show that, for example, when skin is present within a painting, food is less likely to be depicted within the same painting and vice-versa (r = -.59). Next, we selected 15 skilled AMT annotators to annotate more than 100k bounding boxes of the 14 materials. This allowed us to identify the spatial location within paintings where materials are depicted. For example, we find that glass is more often depicted within the top-left quadrant of the painting, which could be related to art-historical literature noting that light is often depicted as originating from the top-left (perhaps through a glass window). In the next step, more detailed material labels will be obtained, for example fabrics will be refined as silk or cotton or wool, etc. This in-depth dataset of material depictions can enable various perceptual, computational and historical analyses that could enable a deeper understanding of material perception and depiction.


  • Open Access

    Painterly depiction of material properties

    by Mitchell J. P. van Zuijlen, Sylvia C. Pont, Maarten W. A. Wijntjes

    Painters are masters of depiction and have learned to evoke a clear perception of materials and material attributes in a natural, three-dimensional setting, with complex lighting conditions. Furthermore, painters are not constrained by reality, meaning that they could paint materials without exactly following the laws of nature, while still evoking the perception of materials. Paintings have to our knowledge not been studied on a big scale from a material perception perspective. In this article, we studied the perception of painted materials and their attributes by using human annotations to find instances of 15 materials, such as wood, stone, fabric, etc. Participants made perceptual judgments about 30 unique segments of these materials for 10 material attributes, such as glossiness, roughness, hardness, etc. We found that participants were able to perform this task well while being highly consistent. Participants, however, did not consistently agree with each other, and the measure of consistency depended on the material attribute being perceived. Additionally, we found that material perception appears to function independently of the medium of depiction—the results of our principal component analysis agreed well with findings in former studies for photographs and computer renderings.