Jens Madsen

When participants are engaged with auditory narratives, physiological and neural signals exhibit temporal correlations between subjects. The intersubject correlation (ISC) increases when attention is directed to the stories, suggesting that shared neural and bodily dynamics arise from a similar processing of the narratives. In this study, we investigated whether the ISC of the EEG elicited by auditory narratives is driven by low-level acoustic (envelope, spectrogram) and/or higher-level linguistic information (word onset, word surprisal) in two groups of healthy participants during passive, attentive and distracted listening. We use temporal response functions (TRFs) for acoustic, and linguistic features to assess the contribution of each feature to the ISC, measured using correlated component analysis (CorrCA). TRFs derived for acoustic features explained a larger fraction of variance in the EEG than linguistic features and were the main contributors to the ISC. The attention-related increase in ISC was driven by all features. Importantly, word surprisal had an effect on ISC only during active story engagement, with timing and scalp distribution consistent with language processing. Notably, the linear responses captured by TRFs only explained a small amount of the overall ISC, suggesting that ISC is largely driven by nonlinear responses to the narratives. Identifying the factors that drive these common neural responses, is clinically relevant for interpreting EEG ISC exhibited in unresponsive patients. We propose that the combined use of ISC and TRFs has the potential to provide meaningful markers of language processing in patients with disorders of consciousness, and we suggest practical recommendations for implementing these approaches in clinical settings.

The human body displays slow, correlated fluctuations in brain activity, autonomic physiology, and incidental behavior. These fluctuations are commonly linked to arousal and cognition, yet it remains unclear whether they reflect task-dependent reconfiguration of brain-body interactions or arise from a stable endogenous dynamic. Here, we used a multivariate dynamical systems approach to analyze simultaneously recorded neural activity, autonomic physiology, and behavior during rest and during attentive and distracted listening to spoken narratives. We found that narrative listening did not substantially alter the endogenous dynamics observed during rest. Instead, peripheral physiology and behavior predicted cortical activity more reliably than the reverse, revealing a pronounced directional asymmetry. In contrast, stimulus-driven physiological responses were primarily mediated by cortical processing rather than direct autonomic drive. Together, these findings indicate that slow brain-body dynamics are stable across cognitive states and provide a structured context through which narratives influence physiology.

The brain, body, and behavior dataset (BBBD) provides multimodal recordings from 178 participants across five experiments in which short educational videos were viewed. The dataset contains about 110 hours of electroencephalogram (EEG), electrooculogram (EOG), electrocardiogram (ECG), respiration, pupil size, gaze position, saccades, blinks, fixations, and head-motion signals, all time-aligned to the video and standardized in the Brain Imaging Data Structure (BIDS) format. Participants watched three to six videos (mean = 28 ± 5 min) under conditions that manipulated attention (attentive versus distracted), learning goals (incidental versus intentional), and motivation (monetary incentive). Demographic data, Adult ADHD Self-Report Scale (ASRS) scores, and digit-span working-memory assessments are included. Technical validation shows expected effects: higher alpha-band power during distraction, a posterior-to-anterior gradient in EEG power, increased blink rate, and reduced saccade rate when attention was diverted. Approximately 97 percent of the continuous signal data are newly released. BBBD enables investigation of how neural, ocular, and physiological dynamics relate to attention and learning during naturalistic video viewing.

We sample visual scenes with short gaze fixations separated by saccades. While low-level integration is known, semantic integration of foveal vision across multiple fixations remains unclear. We hypothesized that the brain responds to changes in semantic information from one fixation to the next, and therefore postulated a neural signal associated with semantic novelty for each saccade. Novelty was measured using a deep network on foveal vision. Novelty modulated frontal and occipital fixation-related potentials in human EEG during natural viewing of full-length movies (3.4×10 ⁶ saccades). Intracranial recordings in humans (9.0×10 ⁴ saccades) and non-human primates (3.3×10 ⁴ saccades) revealed broadband high-frequency activity modulations in ventromedial visual and frontal brain areas. This modulation was stronger for movies than static images, and frontal modulation preceded occipital modulation, suggesting top-down effects. This modulation of fixation-related activity with novelty suggests that foveal representations are integrated across saccades to construct scene representations during natural viewing in primates. Significance statement: Primate vision relies on eye movements to sample the world, yet how the brain integrates high-level meaning across these “snapshots” remains a mystery. By combining deep-learning models with massive electrophysiological datasets from humans and macaques, we show that the brain responds to changes in semantic content between fixations. We identified a “semantic novelty” signal that modulates neural activity across the primate cortex, appearing in frontal areas before a saccade is completed. These findings suggest visual processing is not a series of independent glimpses, but a continuous process integrating foveal information at the semantic level. This work bridges biological vision and artificial intelligence, providing a new framework for understanding how the brain maintains a coherent understanding of a dynamic world.

When participants are engaged with auditory narratives, physiological and neural signals exhibit temporal correlations between subjects. The intersubject correlation (ISC) increases when attention is directed to the stories, suggesting that shared neural and bodily dynamics arise from a similar processing of the narratives. Identifying the factors that drive these common responses is clinically relevant for interpreting EEG ISC exhibited in unresponsive patients. In this study, we investigated whether the ISC of the EEG elicited by auditory narratives is driven by low-level acoustic (envelope, spectrogram) and/or higher-level linguistic information (word onset, word surprisal) in two groups of healthy participants during passive, attentive and distracted listening. We use temporal response functions (TRFs) for acoustic, and linguistic features to assess the contribution of each feature to the ISC, measured using correlated component analysis (CorrCA). TRFs derived for acoustic features explained a larger fraction of variance in the EEG than linguistic features and were the main contributors to the ISC. The attention-related increase in ISC was driven by all features. Importantly, word surprisal had an effect on ISC only during active story engagement, with timing and scalp distribution consistent with language processing. Notably, the linear responses captured by TRFs only explained a small amount of the overall ISC, suggesting that ISC is largely driven by nonlinear responses to the narratives. We propose that the combined use of ISC and TRFs has the potential to provide meaningful markers of language processing in patients with disorders of consciousness, and we suggest practical recommendations for their implementation.

The human body displays slow, spontaneous fluctuations in brain activity, autonomic physiology, and small incidental movements. It is unknown whether these co-fluctuations reflect a stable endogenous brain-body dynamic, or whether this dynamic varies with cognitive state. We addressed this question using a dynamical systems approach to analyze simultaneously recorded neural activity (EEG), autonomic physiology, and behavior while participants listened to spoken narratives or were at rest. We found that cognitive state did not substantially alter the endogenous dynamic. Acoustic and linguistic features predicted neural activity, which in turn affected physiological responses. Only low-level sound fluctuations exerted direct effects on autonomic signals. Peripheral physiology and behavior exerted stronger influences on EEG than the reverse. These findings suggest that slow co-fluctuations are the result of a stablebrain-body dynamic with strong bottom-up feedback, and that the narrative entrains this dynamic by engaging cognition.

Attention-Deficit/Hyperactivity Disorder (ADHD) and Autism Spectrum Disorder (ASD) frequently co-occur with overlapping symptoms. We investigated whether automated behavioral analysis during a clinician-child interview can identify distinct, objective features of the two disorders. Analyzing audio-video recordings of 2,341 youths (ages 5–22) in a broad community sample, multivariate models revealed that language difficulties often attributed to ADHD are primarily explained by age, cognitive ability, or co-occurring ASD. Increased motor activity specifically marked hyperactive-impulsive ADHD, but not ASD or inattentive ADHD. ASD was uniquely characterized by divergent narrative production and perspective-taking, alongside a distinct vocal profile of higher pitch and intensity, despite structurally intact language. While these digital behavioral measures correlate with most diagnostic categories and age, the joint analysis effectively separates the effects of ASD from ADHD. These findings show that scalable digital assessment from recorded clinical interviews can disentangle overlapping ASD and ADHD diagnoses into domain-specific behavioral signatures.

Summary: Poor academic performance is often linked to attention deficits, but it remains unclear whether these reflect enduring traits or momentary lapses in focus. In this study, we examined how state and trait-level attention relate to learning from short educational videos. Across four experiments (N = 152), participants completed standardized assessments of inattention, hyperactivity, working memory capacity, and GPA. They then viewed 3-6-minute educational videos while electroencephalography (EEG) tracked neural responses, followed by quizzes assessing short-term retention. Neural synchrony during video viewing, a measure of attentional state, strongly predicted test performance (p < 0.001). In contrast, trait inattention did not predict retention (p > 0.05), although it was negatively associated with GPA. Attentional state was positively associated with working memory capacity but not with trait inattention. These findings suggest that students with attentional difficulties can still learn effectively from short, engaging content, emphasizing the importance of instructional format in supporting diverse learners.

When humans engage with video, their brain and body interact in response to sensory input. To investigate these interactions, we recorded and are releasing a dataset from N=178 participants across five experiments featuring short online educational videos. This dataset comprises approximately 110 hours of multimodal data including electrocardiogram (ECG), heart rate, respiration, breathing rate, pupil size, electrooculogram (EOG), gaze position, saccades, blinks, fixations, head movement, and electroencephalogram (EEG). Participants viewed 3-6 videos (mean total duration: 28±5 min) to test attentional states (attentive vs. distracted), memory retention (multiple-choice questions), learning scenarios (incidental vs. intentional), and an intervention (monetary incentive). Demographic data, ADHD self-report (ASRS), and working memory assessments (digit span) were collected. Basic statistics and noteworthy effects: increased alpha power in a distracted condition, broadband EEG power increases from posterior to anterior scalp, increased blink-rate, and decreased saccade-rate in distracted and intervention conditions. All modalities are time-aligned with stimuli and standardized using BIDS, making the dataset valuable for researchers investigating attention, memory, and learning in naturalistic settings.