Yueyuan Zheng

Mounting evidence suggests that hierarchical psychopathology factors underlie psychiatric comorbidity. However, the exact neurobiological characterizations of these multilevel factors remain elusive. Here, leveraging the brain-behavior predictive framework with a 10-year longitudinal imaging-genetic cohort (IMAGEN, ages 14, 19 and 23, N = 1,750), we constructed 2 neural factors underlying externalizing and internalizing symptoms, which were reproducible across 6 clinical and population-based datasets (ABCD, STRATIFY/ESTRA, ABIDE II, ADHD-200 and XiNan, from age 10 to age 36, N = 3,765). These two neural factors exhibit distinct neural configurations: hyperconnectivity in impulsivity-related circuits for the externalizing symptoms and hypoconnectivity in goal-directed circuits for the internalizing symptoms. Both factors also differ in their cognitive-behavior relevance, genetic substrates and developmental profiles. Together with previous findings, we propose a hierarchical neurocognitive model of comorbid psychopathology (NeuroHiP) from preadolescence to adulthood, comprising a general neuropsychopathological factor (manifested as inefficient executive control) and two stratified factors of externalizing (deficient inhibition control) and internalizing (impaired goal-directed function) symptoms, respectively. These holistic insights are crucial for the development of stratified therapeutic interventions for mental disorders.

Emotions can greatly influence cognitive processes, such as reading. However, recent studies on the interaction between emotions and reading comprehension provide a murky picture with contradictory claims. Here, we offer one explanation for this phenomenon, that is, the influence of emotion is dependent on different attention goals during reading comprehension. To test this hypothesis, we manipulated question types that were presented to participants prior to reading to evoke different attentional goals. By utilizing an advanced eye movement analysis method Eye Movement with Hidden Markov Models (EMHMM) with co-clustering, we discovered two representative attention patterns from eye fixations: an expanded, globalized attention pattern and a more focused, localized attention pattern. The subsequent analysis showed that globalized pattern led to better comprehension accuracy for summative questions, and localized pattern facilitated detailed question answering. Moreover, negative emotion induces more localized attention for detailed questions only, which could further explain its facilitation effect in detailed question answering. Our findings suggest that the impact of emotions on reading comprehension is shaped by the attention goal of the task. This suggests that emotion appears to guide attention in ways that either support or hinder comprehension depending on what the higher-level goal is.

Semantic priming has been studied for nearly 50 years across various experimental manipulations and theoretical frameworks. Although previous studies provide insight into the cognitive underpinnings of semantic representations, they have suffered from small sample sizes and a lack of linguistic and cultural diversity. In this Registered Report, we measured the size and the variability of the semantic priming effect across 19 languages (n = 25,163 participants analysed) by creating the largest available database of semantic priming values using an adaptive sampling procedure. We found evidence for semantic priming in terms of differences in response latencies between related word-pair conditions and unrelated word-pair conditions. Model comparisons showed that the inclusion of a random intercept for language improved model fit, providing support for variability in semantic priming across languages. This study highlights the robustness and variability of semantic priming across languages and provides a rich, linguistically diverse dataset for further analysis. The Stage 1 protocol for this Registered Report was accepted in principle on 15 July 2022. The protocol, as accepted by the journal, can be found at https://osf.io/u5bp6 (registration) or https://osf.io/q4fjy (preprint version 6, 31 May 2022).

Semantic priming has been studied for nearly 50 years across various experimental manipulations and theoretical frameworks. Although previous studies provide insight into the cognitive underpinnings of semantic representations, they have suffered from small sample sizes and a lack of linguistic and cultural diversity. In this Registered Report, we measured the size and the variability of the semantic priming effect across 19 languages (N = 25,163 participants analyzed) by creating the largest available database of semantic priming values based on an adaptive sampling procedure. We found evidence for semantic priming in terms of differences in response latencies between related word-pair conditions and unrelated word-pair conditions. Model comparisons showed that inclusion of a random intercept for language improved model fit, providing support for variability in semantic priming across languages. This study highlights the robustness and variability of semantic priming across languages and provides a rich, linguistically diverse dataset for further analysis.

Explainable AI (XAI) methods provide explanations of AI models, but our understanding of how they compare with human explanations remains limited. Here, we examined human participants' attention strategies when classifying images and when explaining how they classified the images through eye‐tracking and compared their attention strategies with saliency‐based explanations from current XAI methods. We found that humans adopted more explorative attention strategies for the explanation task than the classification task itself. Two representative explanation strategies were identified through clustering: One involved focused visual scanning on foreground objects with more conceptual explanations, which contained more specific information for inferring class labels, whereas the other involved explorative scanning with more visual explanations, which were rated higher in effectiveness for early category learning. Interestingly, XAI saliency map explanations had the highest similarity to the explorative attention strategy in humans, and explanations highlighting discriminative features from invoking observable causality through perturbation had higher similarity to human strategies than those highlighting internal features associated with higher class score. Thus, humans use both visual and conceptual information during explanation, which serve different purposes, and XAI methods that highlight features informing observable causality match better with human explanations, potentially more accessible to users.

Recent research has suggested that dynamic emotion recognition involves strong audiovisual association; that is, facial or vocal information alone automatically induces perceptual processes in the other modality. We hypothesized that different emotions may differ in the automaticity of audiovisual association, resulting in differential audiovisual information processing. Participants judged the emotion of a talking-head video under audiovisual, video-only (with no sound), and audio-only (with a static neutral face) conditions. Among the six basic emotions, disgust had the largest audiovisual advantage over the unimodal conditions in recognition accuracy. In addition, in the recognition of all the emotions except for disgust, participants' eye-movement patterns did not change significantly across the three conditions, suggesting mandatory audiovisual information processing. In contrast, in disgust recognition, participants' eye movements in the audiovisual condition were less eyes-focused than the video-only condition and more eyes-focused than the audio-only condition, suggesting that audio information in the audiovisual condition interfered with eye-movement planning for important features (eyes) for disgust. In addition, those whose eye-movement pattern was affected less by concurrent disgusted voice information benefited more in recognition accuracy. Disgust recognition is learned later in life and thus may involve a reduced amount of audiovisual associative learning. Consequently, audiovisual association in disgust recognition is less automatic and demands more attentional resources than other emotions. Thus, audiovisual information processing in emotion recognition depends on the automaticity of audiovisual association of the emotion resulting from associative learning. This finding has important implications for real-life emotion recognition and multimodal learning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Greater eyes-focused eye movement pattern during face recognition is associated with better performance in adults but not in children. We test the hypothesis that higher eye movement consistency across trials, instead of a greater eyes-focused pattern, predicts better performance in children since it reflects capacity in developing visual routines. We first simulated visual routine development through combining deep neural network and hidden Markov model that jointly learn perceptual representations and eye movement strategies for face recognition. The model accounted for the advantage of eyes-focused pattern in adults, and predicted that in children (partially trained models) consistency but not pattern of eye movements predicted recognition performance. This result was then verified with data from typically developing children. In addition, lower eye movement consistency in children was associated with autism diagnosis, particularly autistic traits in social skills. Thus, children’s face recognition involves visual routine development through social exposure, indexed by eye movement consistency.

Reading is an essential medium for learning, but it is challenging to measure learners’ cognitive processes during reading. Eye-tracking, as an approach in multimodal learning analytics (MmLA), can provide fine-grained data that reflect cognitive processes during reading. In this study, we investigated whether eye movements could predict passage reading performance in addition to language proficiency and cognitive abilities. In particular, we assessed learners’ eye movement pattern and consistency through a novel method, Eye Movement analysis with Hidden Markov Models (EMHMM), in addition to traditional eye movement measures. We found that longer saccade length predicted faster reading speed Also, higher English proficiency predicted faster reading speed through the mediation of longer saccade length. In contrast, reading comprehension accuracy was best predicted by a more consistent eye fixation at the beginning of reading engagement, which may result from a better developed visual routine due to higher reading expertise. These findings have important implications for ways to assess and facilitate learners’ reading through eye movement measures and to examine factors influencing reading performance. The methods adopted could further the development of MmLA and serve as an empirical example of understanding learners’ cognitive processes through collecting and modeling critical learner-centered metrics in novel modalities.