Amanda J. Haskins

Our visual landscape consists of not only people, places, and objects (e.g., “soldier”, “stadium”, “flag”), but also the conceptual relationships that unite them (e.g., “patriotism”). Because conceptual knowledge varies across individuals, this level of structure may support individualized patterns of attentional selection during naturalistic scene viewing. Here, we ask whether individuals’ gaze patterns reflect, in part, latent attentional priorities organized in conceptual space. Participants (N=61) freely explored a diverse set of immersive real-world scenes (N=100) in head-mounted VR while their gaze position was continuously recorded. We modeled gaze behavior using spatial, visual, and conceptual feature spaces, leveraging embeddings from large vision and language models, to uncover the latent priorities guiding individuals’ unique patterns of selective attention across environments. Individuals exhibited stable and idiosyncratic gaze patterns across scenes and test–retest sessions, consistent with trait-like individual differences in attention. Spatial, visual, and conceptual feature spaces each explain unique variance in individual gaze patterns, with conceptual features contributing variance beyond that explained by spatial and visual features alone. Notably, language-model–based predictions were particularly effective at capturing these individualized patterns. Together, these findings indicate that naturalistic visual attention is structured at multiple levels -- including a conceptual level -- revealing stable individual differences in how people sample and prioritize information across complex visual environments.

How does knowledge affect visual exploration? Here, we investigated whether learning about an object would change participants’ gaze behavior. Adults (N = 22) wore a head-mounted eye tracker while exploring real-world objects that were relatively unfamiliar (e.g., a rolodex). We assessed learning-related differences in both (1) oculomotor behavior (e.g., fixation durations) and (2) participants' fixated visual content by leveraging embeddings from a large vision-language model. We find evidence that before learning, gaze is more exploratory (i.e., shorter, more frequent fixations). Moreover, we find that differences in fixated content across participants are increased after learning about an object, suggesting that knowledge states may contribute to differences in gaze behavior. These results underscore the importance for future work of quantifying individual knowledge states to further leverage naturalistic eye tracking as a window into learning and cognitive development.

Conversation is a dynamic, multi-modal activity involving the exchange of complex streams of information like words, prosody, gesture, eye contact, and backchannels. Understanding how these different channels interact in naturalistic scenarios is essential for understanding the mechanisms governing human communication.Past studies suggested that the duration of words is tied to their predictability in context, but it remains unclear whether this relationship is speaker-oriented (e.g. retrieval or production-based) or due to listener-oriented, intelligibility-based pressures (i.e. emphasizing unpredictable words to ease comprehension).This study aims to examine the relationship between predictability and additional streams of speaker and listener behavior, to test how much intelligibility-oriented principles impact conversation.We use the GPT-2 large language model to assess the relationship between surprisal, a measure of unpredictability, and several variables known to play an important role in conversation --- the prosodic features of duration, pitch, and intensity, and the timing of listener backchannels. We perform this analysis on the CANDOR corpus of naturalistic spoken video call conversation between strangers in English.In keeping with previous results using n-gram predictability, we find that GPT-2 surprisal predicts significantly higher values for duration. Moreover, surprisal also predicts maximum pitch and maximum intensity even when controlling for duration. Additionally, listener backchannels were more likely to overlap high-surprisal words compared to low-surprisal words, suggesting that listeners provide verbal feedback and acknowledgement of unpredictable, i.e. informative, words.The results provide additional support for intelligibility-based accounts, which hold that language production is sensitive to a pressure for successful communication, not just speaker-oriented pressures.

Sensory differences are nearly universal in autism, but their genetic origins are poorly understood. Here, we tested how individuals with an autism-linked genotype, 16p.11.2 deletion (“16p”), attend to visual information in immersive, real-world photospheres. We monitored participants’ (N = 44) gaze while they actively explored 360° scenes via headmounted virtual reality. We modeled the visually salient and semantically meaningful information in scenes and quantified the relative bottom-up vs. top-down influences on attentional deployment. We found, when compared to typically developed control (TD) participants, 16p participants’ attention was less dominantly predicted by semantically meaningful scene regions, relative to visually salient regions. These results suggest that a reduction in top-down relative to bottom-up attention characterizes how individuals with 16p.11.2 deletions engage with naturalistic visual environments.

Sensory differences are nearly universal in autism, but their genetic origins are poorly understood. Here, we tested how individuals with an autism-linked genotype, 16p.11.2 deletion (“16p”), attend to visual information in immersive, real-world photospheres. We monitored participants’ (N = 44) gaze while they actively explored 360° scenes via headmounted virtual reality. We modeled the visually salient and semantically meaningful information in scenes and quantified the relative bottom-up vs. top-down influences on attentional deployment. We found, when compared to typically developed control (TD) participants, 16p participants’ attention was less dominantly predicted by semantically meaningful scene regions, relative to visually salient regions. These results suggest that a relative reduction in top-down relative to bottom-up attention characterizes how individuals with 16p.11.2 deletions engage with naturalistic visual environments.

Individuals with autism spectrum conditions (ASC) describe differences in both social cognition and sensory processing, but little is known about the causal relationship between these disparate functional domains. In the present study, we sought to understand how a core characteristic of autism—reduced social attention—is impacted by the complex multisensory signals present in real‐world environments. We tested the hypothesis that reductions in social attention associated with autism would be magnified by increasing perceptual load (e.g., motion, multisensory cues). Adult participants (N = 40; 19 ASC) explored a diverse set of 360° real‐world scenes in a naturalistic, active viewing paradigm (immersive virtual reality + eyetracking). Across three conditions, we systematically varied perceptual load while holding the social and semantic information present in each scene constant. We demonstrate that reduced social attention is not a static signature of the autistic phenotype. Rather, group differences in social attention emerged with increasing perceptual load in naturalistic environments, and the susceptibility of social attention to perceptual load predicted continuous measures of autistic traits across groups. Crucially, this pattern was specific to the social domain: we did not observe differential impacts of perceptual load on attention directed toward nonsocial semantic (i.e., object, place) information or low‐level fixation behavior (i.e., overall fixation frequency or duration). This study provides a direct link between social and sensory processing in autism. Moreover, reduced social attention may be an inaccurate characterization of autism. Instead, our results suggest that social attention in autism is better explained by “social vulnerability,” particularly to the perceptual load of real‐world environments.

Differences in both social and sensory processing are well described in the autism literature, but little is known about the causal relationship between these domains, particularly in real-world environments. In the present study, we sought to understand how a core characteristic of the autistic phenotype – reduced social attention – is impacted by the complex multisensory signals present in environments like those encountered in daily life. We tested the hypothesis that autistic group differences in social attention would be magnified by increasing perceptual load (e.g., motion, multisensory cues). Adult participants (N = 40; 19 ASC) explored a diverse set of 360° real-world scenes in a naturalistic, active viewing paradigm (immersive virtual reality + eyetracking). Across three conditions, we systematically varied perceptual load while holding the social and semantic information present in each scene constant. We demonstrate that group differences in social attention are not a static signature of the autistic phenotype; rather, they emerge with increasing perceptual load in naturalistic environments. Crucially, this pattern was specific to the social domain and not driven by low-level oculomotor differences: we did not observe differential impacts of perceptual load on attention directed toward nonsocial semantic (i.e., object, place) information or fixation behavior. This study provides a direct link between social and sensory processing in autism. Moreover, reduced social attention may be an inaccurate characterization of autism. Instead, our results suggest that social attention in autism is better explained by “social vulnerability", particularly to the perceptual load of real-world environments.

How do we construct a sense of place in a real-world environment? Real-world environments are actively explored via saccades, head turns, and body movements. Yet, little is known about how humans process real-world scene information during active viewing conditions. Here, we exploited recent developments in virtual reality (VR) and in-headset eye-tracking to test the impact of active vs. passive viewing conditions on gaze behavior while participants explored novel, real-world, 360° scenes. In one condition, participants actively explored 360° photospheres from a first-person perspective via self-directed motion (saccades and head turns). In another condition, photospheres were passively displayed to participants while they were head-restricted. We found that, relative to passive viewers, active viewers displayed increased attention to semantically meaningful scene regions, suggesting more exploratory, information-seeking gaze behavior. We also observed signatures of exploratory behavior in eye movements, such as quicker, more entropic fixations during active as compared with passive viewing conditions. These results show that active viewing influences every aspect of gaze behavior, from the way we move our eyes to what we choose to attend to. Moreover, these results offer key benchmark measurements of gaze behavior in 360°, naturalistic environments.

Vision is an active process. Humans actively sample their sensory environment via saccades, head turns, and body movements. Yet, little is known about active visual processing in real-world environments. Here, we exploited recent advances in immersive virtual reality (VR) and in-headset eye-tracking to show that active viewing conditions impact how humans process complex, real-world scenes. Specifically, we used quantitative, model-based analyses to compare which visual features participants prioritize over others while encoding a novel environment in two experimental conditions: active and passive. In the active condition, participants used head-mounted VR displays to explore 360º scenes from a first-person perspective via self-directed motion (saccades and head turns). In the passive condition, 360º scenes were passively displayed to participants within the VR headset while they were head-restricted. Our results show that signatures of top-down attentional guidance increase in active viewing conditions: active viewers disproportionately allocate their attention to semantically relevant scene features, as compared with passive viewers. We also observed increased signatures of exploratory behavior in eye movements, such as quicker, more entropic fixations during active as compared with passive viewing conditions. These results have broad implications for studies of visual cognition, suggesting that active viewing influences every aspect of gaze behavior – from the way we move our eyes to what we choose to attend to – as we construct a sense of place in a real-world environment. Significance Statement: Eye-tracking in immersive virtual reality offers an unprecedented opportunity to study human gaze behavior under naturalistic viewing conditions without sacrificing experimental control. Here, we advanced this new technique to show how humans deploy attention as they encode a diverse set of 360º, real-world scenes, actively explored from a first-person perspective using head turns and saccades. Our results build on classic studies in psychology, showing that active, as compared with passive, viewing conditions fundamentally alter perceptual processing. Specifically, active viewing conditions increase information-seeking behavior in humans, producing faster, more entropic fixations, which are disproportionately deployed to scene areas that are rich in semantic meaning. In addition, our results offer key benchmark measurements of gaze behavior in 360°, naturalistic environments.