Elena Kushnerenko

Research on audiovisual speech integration has reported high levels of individual variability, especially among young infants. In the present study we tested the hypothesis that this variability results from individual differences in the maturation of audiovisual speech processing during infancy. A developmental shift in selective attention to audiovisual speech has been demonstrated between 6 and 9 months with an increase in the time spent looking to articulating mouths as compared to eyes (Lewkowicz & Hansen-Tift. (2012) Proc. Natl Acad. Sci. USA, 109, 1431-1436; Tomalski et al. (2012) Eur. J. Dev. Psychol., 1-14). In the present study we tested whether these changes in behavioural maturational level are associated with differences in brain responses to audiovisual speech across this age range. We measured high-density event-related potentials (ERPs) in response to videos of audiovisually matching and mismatched syllables /ba/ and /ga/, and subsequently examined visual scanning of the same stimuli with eye-tracking. There were no clear age-specific changes in ERPs, but the amplitude of audiovisual mismatch response (AVMMR) to the combination of visual /ba/ and auditory /ga/ was strongly negatively associated with looking time to the mouth in the same condition. These results have significant implications for our understanding of individual differences in neural signatures of audiovisual speech processing in infants, suggesting that they are not strictly related to chronological age but instead associated with the maturation of looking behaviour, and develop at individual rates in the second half of the first year of life

Orienting to salient events in the environment is a first step in the development of attention in young infants. Electrophysiological studies have indicated that in newborns and young infants, sounds with widely distributed spectral energy, such as noise and various environmental sounds, as well as sounds widely deviating from their context elicit an event related potential (ERP) similar to the adult P3a response. We discuss how the maturation of event-related potentials parallels the process of the development of passive auditory attention during the first year of life. Behavioural studies have indicated that the neonatal orientation to high energy stimuli gradually changes to attending to genuine novelty and other significant events by approximately 9 months of age. In accordance with these changes, in newborns, the ERP response to large acoustic deviance is dramatically larger than that to small and moderate deviations. This ERP difference, however, rapidly decreases within first months of life and the differentiation of the ERP response to genuine novelty from that to spectrally rich but repeatedly presented sounds commences during the same period. The relative decrease of the response amplitudes elicited by high energy stimuli may reflect development of an inhibitory brain network suppressing the processing of uninformative stimuli. Based on data obtained from healthy full term and pre term infants as well as from infants at risk for various developmental problems, we suggest that the electrophysiological indices of the processing of acoustic and contextual deviance may be indicative of the functioning of auditory attention, a crucial prerequisite of learning and language development.

The use of visual cues during the processing of audiovisual (AV) speech is known to be less efficient in children and adults with language difficulties and difficulties are known to be more prevalent in children from low-income populations. In the present study, we followed an economically diverse group of thirty-seven infants longitudinally from 6–9 months to 14–16 months of age. We used eye-tracking to examine whether individual differences in visual attention during AV processing of speech in 6–9 month old infants, particularly when processing congruent and incongruent auditory and visual speech cues, might be indicative of their later language development. Twenty-two of these 6–9 month old infants also participated in an event-related potential (ERP) AV task within the same experimental session. Language development was then followed-up at the age of 14–16 months, using two measures of language development, the Preschool Language Scale and the Oxford Communicative Development Inventory. The results show that those infants who were less efficient in auditory speech processing at the age of 6–9 months had lower receptive language scores at 14–16 months. A correlational analysis revealed that the pattern of face scanning and ERP responses to audiovisually incongruent stimuli at 6–9 months were both significantly associated with language development at 14–16 months. These findings add to the understanding of individual differences in neural signatures of AV processing and associated looking behavior in infants.

Inconsistency in mismatch component polarity in infants has been most consistently related to maturational level, though several other factors, such as magnitude of auditory change (difficulty of the contrast) and methodological issues (high-pass filtering) have also been discussed. The mismatch response (MMR) follows the maturation of obligatory auditory event-related potential (ERP): a broad negativity in preterm and immature newborns turns into a predominantly positive response at the same time as positive ERP components grow during first months of life. The next step in maturation is the development of the N250 during the second half of the first year. Correspondingly, the early negative MMR starts to be observed at the same latency. This early negative discriminative response can be observed even earlier in development in response to a robust spectral change in stimuli, but not to more fine acoustic contrasts (small frequency changes or changes in speech syllables). Some studies employing very large spectral change (white-noise deviants or harmonic tones with 50 % pitch change) reported early negativity even in newborns. In this study, the MMRs were elicited in 20-24 month old infants in four experimental conditions: a) large frequency change - harmonic tones, standard 500 Hz FF, deviant 750 Hz; b) small CV change (deviant /ga/ and standard /ka/) c) large duration change (harmonic tones, standard 200 ms, deviant 100 ms) d) control condition. In the control condition, the 750-Hz ‘deviant’ was presented randomly and equiprobably among two other tones (500 and 625 Hz FF). This was performed in order to compare responses to deviant and to an identical stimulus presented at a comparable presentation rate to avoid adaptation if presented repetitively or orienting if presented against silent background. The results showed that, the ERP to deviant was significantly more negative than that to standard in response to the large frequency change (F(1,11)=15.64; p<.0023), but more positive to syllable (F(1,11)=10.41; p<.0081) and duration changes (F(1,11)=6.47; p<.0273) within the same latency window (200-300 ms from change onset). There were no significant differences between the N250 amplitude of the frequency deviant and of identical tone presented in a control condition. This suggests that while the positive MMR appears to be more general response to auditory violation in infants, the fast early negative one (<250 ms) might be specific to large frequency changes irrespective of whether they violate auditory expectations or not and thus, may not represent an adult-like MMN.

How effortlessly and quickly infants acquire their native language remains one of the most intriguing questions of human development. Our study extends this question into the audiovisual domain, taking into consideration visual speech cues, which were recently shown to have more importance for young infants than previously anticipated [Weikum WM, Vouloumanos A, Navarra J, Soto-Faraco S, Sebastián-Gallés N, Werker JF (2007) Science 316:1159]. A particularly interesting phenomenon of audiovisual speech perception is the McGurk effect [McGurk H, MacDonald J (1976) Nature 264:746–748], an illusory speech percept resulting from integration of incongruent auditory and visual speech cues. For some phonemes, the human brain does not detect the mismatch between conflicting auditory and visual cues but automatically assimilates them into the closest legal phoneme, sometimes different from both auditory and visual ones. Measuring event-related brain potentials in 5-month-old infants, we demonstrate differential brain responses when conflicting auditory and visual speech cues can be integrated and when they cannot be fused into a single percept. This finding reveals a surprisingly early ability to perceive speech cross-modally and highlights the role of visual speech experience during early postnatal development in learning of the phonemes and phonotactics of the native language.