Janet Huiwen Hsiao

Significant progress has been made in the improvement and downstream applications of the Contrastive Language-Image Pre-training (CLIP) vision-language model, while less attention has been paid to the interpretation of CLIP. We propose a Gradient-based visual and textual Explanation method for CLIP (Grad-ECLIP), which interprets the matching result of CLIP for a specific input image-text pair. By decomposing the encoder's architecture and identifying the relationship between matching similarity and intermediate spatial features, Grad-ECLIP generates effective heat maps that reveal the impact of image regions or words on the CLIP results. Unlike previous Transformer interpretation methods that focus on utilizing self-attention maps, which are typically extremely sparse in CLIP, we produce high-quality visual explanations by applying channel and spatial weights to token features. Qualitative and quantitative evaluations verify the effectiveness and superiority of Grad-ECLIP compared with the state-of-the-art methods. Finally, a series of analyses are conducted based on our visual and textual explanation results, from which we explore the working mechanism of image-text matching, the strengths and limitations in attribution identification of CLIP, and the relationship between the concreteness/abstractness of a word and its usage in CLIP. The code of Grad-ECLIP is available here: https://github.com/Cyang-Zhao/Grad-Eclip.

Although faces are homogeneous stimuli, individuals adopt different eye‐movement patterns when looking at them. Do such differences in eye movements reflect distinct patterns of neural processing of faces? We conducted two experiments to investigate how face‐viewing preferences affect face‐sensitive event‐related potentials (ERPs). In both experiments, participants were separated into two groups: an upper‐focused group who fixated mainly on the eyes of faces and a lower‐focused group who fixated more on the nose and mouth. Participants then completed two fixation number‐controlled blocks in Experiment 1 or two fixation duration‐controlled blocks in Experiment 2, in which the initial fixation location was restricted to the left eye or the nose. In both fixation number‐controlled and duration‐controlled blocks, the upper‐focused group performed better and elicited a larger P1‐to‐N170 peak‐to‐peak amplitude for the eye‐fixation condition than the nose‐fixation condition. In contrast, the lower‐focused group performed evenly and elicited comparable P1‐to‐N170 amplitudes between fixation conditions. In addition, the P1 component was larger for the nose‐fixation condition than the eye‐fixation condition, and this difference was bigger for the lower‐focused group than the upper‐focused group. On the other hand, the P1 appeared to have no relation to behavioral performance. These findings show how individual differences in fixation patterns for faces map to differences in neural processing of those faces, and in turn provide evidence for the intricate relationship between sensory and neural processing of visual information.

Recent research has shown that as compared with neurotypicals, autistic individuals have lower eye movement consistency in face recognition in addition to poorer performance. This effect is associated with Autism-Spectrum Quotient (AQ) Social Skill, suggesting that it may be related to insufficient social exposure for visual routine development. Here, we aimed to investigate whether this phenomenon could also be observed in visual search tasks with social stimuli or with high scene complexity to understand its generalizability and underlying mechanisms. We compared the performance and eye movement behavior of 33 autistic individuals with matched neurotypicals when searching for either social (human) or non-social (vehicle) stimuli in either low complexity or high complexity real-world scenes. The results showed that they had similar hit rate and precision as well as eye movement pattern and consistency. However, autistic individuals had longer search time and made a larger number of fixations with longer fixation durations, suggesting lower efficiency in identifying potential targets. The longer search time and larger number of fixations were both uniquely predicted by AQ Details/Patterns beyond associated cognitive ability measures, suggesting their relevance to autistic traits of restrictive and repetitive behavior. This difficulty was not limited to social stimuli or complex scenes, supporting a domain-general view of deficits in autism spectrum disorder (ASD). Together with findings from face recognition, our results suggest that the manifestation of autistic individuals’ processing difficulties depends on the task demands and related traits. Our findings have important implications for the behavioral markers and core mechanisms of ASD.

Face recognition involves a prolonged developmental process where individuals gradually develop a more consistent yet idiosyncratic visual routine with increasing age. In contrast to this early developmental trend, among 301 middle and older participants, we observed that age was associated with lower eye movement consistency, suggesting an inverted-U shape across the lifespan, and consistency could partially account for the association between age and declined face recognition performance. Also, a less eyes-focused eye movement pattern predicted poorer recognition performance together with lower consistency, suggesting that the idiosyncratic eye movement patterns acquired during early adulthood continue to contribute significantly. Additionally, lower eye movement consistency was associated with poorer selective attention and inhibitory control, indicating that the decline in consistency may reflect difficulties in executing learned visual routines. These findings have important implications for understanding cognitive aging and potential interventions to delay its process through training in face recognition and related visual strategies.

This pilot study evaluated a personalized sleep intervention incorporating motivational interviewing techniques to address sleep barriers, along with tailored sleep hygiene and extension for chronic short sleepers. Eleven university students completed a 14‐day sequential intervention (baseline, sleep hygiene and a combined phase adding 90‐min extension), assessed via actigraphy, daily diaries, ecological momentary assessments and qualitative interviews. The intervention was highly feasible. Total sleep time increased from 5.01 h (baseline) to 5.62 h (sleep hygiene) and 6.67 h (combined phase), alongside reduced bedtime procrastination and improved sleep hygiene practices. Multilevel modelling suggested that sleep hygiene increased time in bed (+0.72 h) and morning vitality, while sleep extension further extended time in bed (+1.49 h) and total sleep time (+0.55 h). Emotional benefits were linked to within‐person increases in sleep duration. Qualitative findings highlighted heterogeneous barriers and the importance of context‐based personalization. These preliminary results support the potential of personalized sleep interventions to improve sleep and emotional health in chronic short sleepers, warranting further controlled trials.

Many AI firms aim to develop AI to benefit humanity by emphasizing fairness and alignment with human values. This study examined potential value alignment bias in Large Language Models (LLMs) by benchmarking them against the World Values Survey dataset. We found that survey responses from state-of-the-art LLMs, GPT-4o and Llama-3.1-70B-Instruct, were better aligned with either Western or richer populations, rather than only Western ones as previously assumed. While prompting the models to act as individuals from the corresponding populations improved the value alignment in general for both models, it helped mitigate the bias towards Western and richer populations only in Llama-3.1-70B-Instruct. The explainable AI method Grad-ECLIP revealed that the open-source model Llama-3.1-70B-Instructs alignment improvement was associated with change in its information use based on the importance of each token in the question text. This validation analysis could not be performed on closed-source models such as GPT-4o. Clustering analyses showed similar alignment profiles from the two models for well-represented populations, whereas the supposedly more capable GPT-4o had lower alignment with underrepresented groups, suggesting a fairness-accuracy trade-off. Among the underrepresented groups, GPT-4o aligned better with Latin American populations, which could result from fine-tuning practices to align with the firms target users. To translate these insights into actionable policy, we propose the Alignment Disclosure Report as a standard for firms to disclose value alignment information so that users, especially those from underrepresented groups, may make informed choices about their use of AI products.

While people are often experts in perceiving and categorizing faces into meaningful social categories (i.e., race), there are suboptimal scenarios such as mask use that may impair face processing. Here we examined how mask use may differentially impact own- and other-race face processing in social categorization, and the underlying neurocognitive mechanisms using simultaneous eye movement and EEG recording. We found that mask use made participants’ face scanning patterns more eyes-focused and consistent, and reduced the differences in both eye movement pattern and early attention-related ERP component P1 between viewing own- and other-race faces. Moreover, mask use did not change how people categorize biracial morphed faces, or the advantage in categorization speed of other-race faces. These results suggest that when perceiving masked faces, information from the eye region may be sufficient for social categorization, and that race-based social categorizations can be impervious to mask use. Interestingly, we found that when viewing other-race faces, where people have less perceptual expertise, those who show more consistent face scanning patterns have more efficient processing of masked faces. These findings have important implications for cross-race face perception, especially when face perception condition becomes suboptimal. Significance statement: As mask use has become a common practice in response to respiratory virus outbreaks, it has inadvertently altered both health practices and the complex dynamics of social interaction. In a world that values diversity and cross-racial interactions, understanding how masks influence our cognitive processes during cross-race face perception is not just timely but vital. Given this context, we examined the effect of mask use on race categorization, by systematically investigating eye movement behavior, and neural representations of own versus other-race faces, and how these mask-induced changes are associated with each other. By utilizing simultaneous eye movement and EEG recording, our study reveals that the eye region can significantly influence social categorization, suggesting that race-based categorizations persist even in the presence of masks. Interestingly, we found that for other-race faces with which people have less perceptual expertise, those who adjust to a more consistent face scanning pattern for masked faces have more efficient processing of masked faces. This highlights the importance of individuals’ visual routine adaptability when the viewing condition is not optimal. Though the current research is called by the demand for COVID-19, our findings can be generalized to a broader context and enhance our understanding of human visual and social cognition.

With the advances in AI technology, comparison studies between humans and AI can not only enhance our understanding of information processing mechanisms underlying human cognition but also facilitate our understanding of AI systems' behaviour and interactions with humans. In particular, explainable AI (XAI) methods, including both computational and experimental methods, can be used to reveal the mechanisms underlying AI's behaviour and its interactions with humans. This information can be used (1) as computational models to study human behaviour, (2) for updating users' beliefs about AI during the interactions, and (3) for evaluation purposes to examine potential ethical issues associated with AI adoption. Different AI systems may require different XAI methods to accurately reveal their underlying mechanisms to facilitate the comparisons with humans. Thus, an important future research direction is to develop task‐specific XAI methods through interdisciplinary approaches across psychology and AI to benefit both psychological research and the development of ethical AI.