NMNancy L. McElwain

Nancy L. McElwain

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Nancy L. McElwain has a total of 42 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Automated Analysis of Naturalistic Recordings in Early Childhood: Applications, challenges, and opportunities

    by Jialu Li, Marvin Lavechin, Xulin Fan, Nancy L. McElwain, Alejandrina Cristia, Paola Garcia-Perera, Mark A. Hasegawa-Johnson

    Naturalistic recordings capture audio in real-world environments where participants behave naturally without interference from researchers or experimental protocols. Naturalistic long-form recordings extend this concept by capturing spontaneous and continuous interactions over extended periods, often spanning hours or even days, in participants’ daily lives. Naturalistic recordings have been extensively used to study children’s behaviors, including how they interact with others in their environment, in the fields of psychology, education, cognitive science, and clinical research. These recordings provide an unobtrusive way to observe children in real-world settings beyond controlled and constrained experimental environments. Advancements in speech technology and machine learning (ML) have provided an initial step for researchers to automatically and systematically analyze large-scale naturalistic recordings of children. Despite the imperfect accuracy of ML models, these tools still offer valuable opportunities to uncover important insights into children’s cognitive and social development. Several critical speech technologies involved include speaker diarization, vocalization classification, word count estimate from adults, speaker verification, and language diarization for code switching. Most of these technologies have been primarily developed for adults, and speech technologies applied to children specifically are still vastly underexplored. To fill this gap, we discuss current progress, challenges, and opportunities in advancing these technologies to analyze naturalistic recordings of children during early development (<3 years of age). We strive to inspire the signal processing community and foster interdisciplinary collaborations to further develop this emerging technology and address its unique challenges and opportunities.


  • Open Access

    Automated Analysis of Naturalistic Recordings in Early Childhood: Applications, Challenges, and Opportunities

    by Jialu Li, Marvin Lavechin, Xulin Fan, Nancy L. McElwain, Alejandrina Cristia, Paola Garcia-Perera, Mark Hasegawa-Johnson

    Naturalistic recordings capture audio in real-world environments where participants behave naturally without interference from researchers or experimental protocols. Naturalistic long-form recordings extend this concept by capturing spontaneous and continuous interactions over extended periods, often spanning hours or even days, in participants' daily lives. Naturalistic recordings have been extensively used to study children's behaviors, including how they interact with others in their environment, in the fields of psychology, education, cognitive science, and clinical research. These recordings provide an unobtrusive way to observe children in real-world settings beyond controlled and constrained experimental environments. Advancements in speech technology and machine learning have provided an initial step for researchers to automatically and systematically analyze large-scale naturalistic recordings of children. Despite the imperfect accuracy of machine learning models, these tools still offer valuable opportunities to uncover important insights into children's cognitive and social development. Several critical speech technologies involved include speaker diarization, vocalization classification, word count estimate from adults, speaker verification, and language diarization for code-switching. Most of these technologies have been primarily developed for adults, and speech technologies applied to children specifically are still vastly under-explored. To fill this gap, we discuss current progress, challenges, and opportunities in advancing these technologies to analyze naturalistic recordings of children during early development (<3 years of age). We strive to inspire the signal processing community and foster interdisciplinary collaborations to further develop this emerging technology and address its unique challenges and opportunities.

  • Open Access

    Evaluating Users’ Experiences of a Child Multimodal Wearable Device: Mixed Methods Approach

    by Nancy L McElwain, Meghan C Fisher, Camille Nebeker, Jordan M Bodway, Bashima Islam, Mark Hasegawa-Johnson

    Background: Wearable devices permit the continuous, unobtrusive collection of data from children in their natural environments and can transform our understanding of child development. Although the use of wearable devices has begun to emerge in research involving children, few studies have considered families’ experiences and perspectives of participating in research of this kind. Objective: Through a mixed methods approach, we assessed parents’ and children’s experiences of using a new wearable device in the home environment. The wearable device was designed specifically for use with infants and young children, and it integrates audio, electrocardiogram, and motion sensors. Methods: In study 1, semistructured phone interviews were conducted with 42 parents of children aged 1 month to 9.5 years who completed 2 day-long recordings using the device, which the children wore on a specially designed shirt. In study 2, a total of 110 parents of children aged 2 months to 5.5 years responded to a questionnaire assessing their experience of completing 3 day-long device recordings in the home. Guided by the Digital Health Checklist, we assessed parental responses from both studies in relation to the following three key domains: (1) access and usability, (2) privacy, and (3) risks and benefits. Results: In study 1, most parents viewed the device as easy to use and safe and remote visits as convenient. Parents’ views on privacy related to the audio recordings were more varied. The use of machine learning algorithms (vs human annotators) in the analysis of the audio data, the ability to stop recordings at any time, and the view that the recordings reflected ordinary family life were some reasons cited by parents who expressed minimal, if any, privacy concerns. Varied risks and benefits were also reported, including perceived child comfort or discomfort, the need to adjust routines to accommodate the study, the understanding gained from the study procedures, and the parent’s and child’s enjoyment of study participation. In study 2, parents’ ratings on 5 close-ended items yielded a similar pattern of findings. Compared with a “neutral” rating, parents were significantly more likely to agree that (1) device instructions were helpful and clear (t109=−45.98; P<.001), (2) they felt comfortable putting the device on their child (t109=−22.22; P<.001), and (3) they felt their child was safe while wearing the device (t109=−34.48; P<.001). They were also less likely to worry about the audio recordings gathered by the device (t108=6.14; P<.001), whereas parents’ rating of the burden of the study procedures did not differ significantly from a “neutral” rating (t109=−0.16; P=.87). Conclusions: On the basis of parents’ feedback, several concrete changes can be implemented to improve this new wearable platform and, ultimately, parents’ and children’s experiences of using child wearable devices in the home setting.


  • Open Access

    Preliminary Technical Validation of LittleBeats™: A Multimodal Sensing Platform to Capture Cardiac Physiology, Motion, and Vocalizations

    by Bashima Islam, Nancy L. McElwain, Jialu Li, Maria I. Davila, Yannan Hu, Kexin Hu, Jordan M. Bodway, Ashutosh Dhekne, Romit Roy Choudhury, Mark Hasegawa-Johnson

    Across five studies, we present the preliminary technical validation of an infant-wearable platform, LittleBeats™, that integrates electrocardiogram (ECG), inertial measurement unit (IMU), and audio sensors. Each sensor modality is validated against data from gold-standard equipment using established algorithms and laboratory tasks. Interbeat interval (IBI) data obtained from the LittleBeats™ ECG sensor indicate acceptable mean absolute percent error rates for both adults (Study 1, N = 16) and infants (Study 2, N = 5) across low- and high-challenge sessions and expected patterns of change in respiratory sinus arrythmia (RSA). For automated activity recognition (upright vs. walk vs. glide vs. squat) using accelerometer data from the LittleBeats™ IMU (Study 3, N = 12 adults), performance was good to excellent, with smartphone (industry standard) data outperforming LittleBeats™ by less than 4 percentage points. Speech emotion recognition (Study 4, N = 8 adults) applied to LittleBeats™ versus smartphone audio data indicated a comparable performance, with no significant difference in error rates. On an automatic speech recognition task (Study 5, N = 12 adults), the best performing algorithm yielded relatively low word error rates, although LittleBeats™ (4.16%) versus smartphone (2.73%) error rates were somewhat higher. Together, these validation studies indicate that LittleBeats™ sensors yield a data quality that is largely comparable to those obtained from gold-standard devices and established protocols used in prior research.


  • Open Access

    Preliminary Technical Validation of LittleBeats™: A Multimodal Sensing Platform to Capture Cardiac Physiology, Motion, and Vocalizations

    by Bashima Islam, Nancy L McElwain, Jialu Li, Maria Davila, Yannan Hu, Kexin Hu, Jordan M Bodway, Ashutosh M Dhekne, Romit Roy Choudhury, Mark Hasegawa-Johnson

    Background: The use of wearable devices has burgeoned over the past decade, including wearables for infants and young children. Typically, such devices assess a single modality, and few have undergone scientific validation. To address this gap, we developed an infant wearable platform, LittleBeats™, that integrates electrocardiogram (ECG), motion, and audio sensors on a single printed circuit board to permit daylong remote assessments of infants and caregivers in home environments. Objective: Complementing our prior reports focused on LittleBeats™ data collected among infants and children in the home context, our main objective here is to present a technical validation of each sensor modality against established laboratory protocols and gold-standard equipment. Materials and Methods: We conducted five studies, including assessments of (a) interbeat interval (IBI) data obtained from the LittleBeats™ ECG sensor versus the gold-standard BIOPAC among adults (Study 1, N=16) and infants (Study 2, N=5), (b) performance of automated activity recognition (upright vs. walk vs. glide vs. squat ) among adults using accelerometer data obtained from LittleBeats™ versus Google Pixel 1 smartphone (Study 3, N=12), and (c) performance of speech emotion recognition (SER; Study 4, N=8) and automatic speech recognition (ASR; Study 5, N =12) algorithms among adult samples using audio data from LittleBeats™ versus smartphone. Results: Results for IBI data obtained from the LittleBeats™ ECG sensor indicate acceptable mean absolute percent error (MAPE) rates for both the adults (MAPE = 5.29% to 5.97%; Study 1) and infants (MAPE =0.96% to 1.66%; Study 2) across low- and high-challenge sessions, as well as expected patterns of change in respiratory sinus arrythmia (RSA) across sessions. For activity recognition (Study 3), LittleBeats™ data showed good to excellent performance (Accuracy = 89%, F1-score=88%; Cohen’s kappa =.79), although the smartphone outperformed LittleBeats™ by less than 4%. Finally, performance on the SER applied to LittleBeats™ versus smartphone audio data (Study 4) indicated comparable performance, with a matched-pairs test indicating no significant difference in error rates (p=.26). On the ASR task (Study 5), the best performing algorithm yielded relatively low word error rates, although LittleBeats™ (4.16%) versus smartphone (2.73%) error rates are somewhat higher. Discussion: Taken together, results from these controlled laboratory studies conducted predominantly with adults indicate that the LittleBeats™ sensors yield data quality that is largely comparable to those obtained from gold-standard devices and protocols that have been used extensively in prior research. The advantages of the LittleBeats™ platform is the (a) integration of multiple sensors into one platform that was (b) designed specifically for use with infants and young children. Leveraging data from the multiple modalities and fine-tuning postprocessing steps to further increase data quality, combined with assessing data quality among infants and young children in their natural environments, will be key next steps in this research program.

  • Open Access

    Classification of Infant Sleep/Wake States: Cross-Attention among Large Scale Pretrained Transformer Networks using Audio, ECG, and IMU Data

    by Kai Chieh Chang, Mark Hasegawa-Johnson, Nancy L. McElwain, Bashima Islam

    Infant sleep is critical to brain and behavioral development. Prior studies on infant sleep/wake classification have been largely limited to reliance on expensive and burdensome polysomnography (PSG) tests in the laboratory or wearable devices that collect single-modality data. To facilitate data collection and accuracy of detection, we aimed to advance this field of study by using a multi-modal wearable device, LittleBeats (LB), to collect audio, electrocardiogram (ECG), and inertial measurement unit (IMU) data among a cohort of 28 infants. We employed a 3-branch (audio/ECG/IMU) large scale transformer-based neural network (NN) to demonstrate the potential of such multi-modal data. We pretrained each branch independently with its respective modality, then finetuned the model by fusing the pretrained transformer layers with cross-attention. We show that multi-modal data significantly improves sleep/wake classification (accuracy = 0.880), compared with use of a single modality (accuracy = 0.732). Our approach to multi-modal mid-level fusion may be adaptable to a diverse range of architectures and tasks, expanding future directions of infant behavioral research.

  • Open Access

    Evaluating Users’ Experiences of a Child Multimodal Wearable Device: Mixed Methods Approach (Preprint)

    by Nancy L McElwain, Meghan C Fisher, Camille Nebeker, Jordan M Bodway, Bashima Islam, Mark Hasegawa-Johnson

    Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Wearable devices permit the continuous, unobtrusive collection of data from children in their natural environments and can transform our understanding of child development. Although the use of wearable devices has begun to emerge in research involving children, few studies have considered families’ experiences and perspectives of participating in research of this kind. Objective: Through a mixed methods approach, we assessed parents’ and children’s experiences of using a new wearable device in the home environment. The wearable device was designed specifically for use with infants and young children, and it integrates audio, electrocardiogram, and motion sensors. Methods: In study 1, semistructured phone interviews were conducted with 42 parents of children aged 1 month to 9.5 years who completed 2 day-long recordings using the device, which the children wore on a specially designed shirt. In study 2, a total of 110 parents of children aged 2 months to 5.5 years responded to a questionnaire assessing their experience of completing 3 day-long device recordings in the home. Guided by the Digital Health Checklist, we assessed parental responses from both studies in relation to the following three key domains: (1) access and usability, (2) privacy, and (3) risks and benefits. Results: In study 1, most parents viewed the device as easy to use and safe and remote visits as convenient. Parents’ views on privacy related to the audio recordings were more varied. The use of machine learning algorithms (vs human annotators) in the analysis of the audio data, the ability to stop recordings at any time, and the view that the recordings reflected ordinary family life were some reasons cited by parents who expressed minimal, if any, privacy concerns. Varied risks and benefits were also reported, including perceived child comfort or discomfort, the need to adjust routines to accommodate the study, the understanding gained from the study procedures, and the parent’s and child’s enjoyment of study participation. In study 2, parents’ ratings on 5 close-ended items yielded a similar pattern of findings. Compared with a “neutral” rating, parents were significantly more likely to agree that (1) device instructions were helpful and clear (t109=−45.98; P<.001), (2) they felt comfortable putting the device on their child (t109=−22.22; P<.001), and (3) they felt their child was safe while wearing the device (t109=−34.48; P<.001). They were also less likely to worry about the audio recordings gathered by the device (t108=6.14; P<.001), whereas parents’ rating of the burden of the study procedures did not differ significantly from a “neutral” rating (t109=−0.16; P=.87). Conclusions: On the basis of parents’ feedback, several concrete changes can be implemented to improve this new wearable platform and, ultimately, parents’ and children’s experiences of using child wearable devices in the home setting.

  • Interactive Contribution of Observed Mother-Youth Emotional Climate and Youth Physiology: A Biopsychosocial Approach to Understanding Youth Coping With Peer Stress

    by Xiaomei Li, Kelly M. Tu, Nancy L. McElwain

    We examined the prospective associations between two indicators of mother-youth emotional climate (i.e., maternal affect, dyadic cohesiveness) during a peer problem discussion and youth coping with peer stress following entry into middle school. We also investigated youth baseline respiratory sinus arrhythmia (RSA) as a moderator of these associations given its role in physiological stress regulation. Participants were 100 mother-youth dyads (53% boys; Mage = 11.05 years; 43% ethnic minorities). More positive maternal affect at Time 1 (spring of 5th grade) was associated with more primary and secondary control engagement coping and advice-seeking at Time 2 (fall of 6th grade); greater dyadic cohesiveness was prospectively linked with more advice-seeking only. Further, the positive association between maternal affect and primary control coping was only evident for youth who showed lower (not higher) baseline RSA. We discuss implications for jointly considering the mother-youth relational context and youth physiological regulatory capacity towards understanding coping socialization.


  • Open Access

    Visualizations of Complex Sequences of Family-Infant Vocalizations Using Bag-of-Audio-Words Approach Based on Wav2vec 2.0 Features

    by Jialu Li, Mark Hasegawa-Johnson, Nancy L. McElwain

    In the U.S., approximately 15-17% of children 2-8 years of age are estimated to have at least one diagnosed mental, behavioral or developmental disorder. However, such disorders often go undiagnosed, and the ability to evaluate and treat disorders in the first years of life is limited. To analyze infant developmental changes, previous studies have shown advanced ML models excel at classifying infant and/or parent vocalizations collected using cell phone, video, or audio-only recording device like LENA. In this study, we pilot test the audio component of a new infant wearable multi-modal device that we have developed called LittleBeats (LB). LB audio pipeline is advanced in that it provides reliable labels for both speaker diarization and vocalization classification tasks, compared with other platforms that only record audio and/or provide speaker diarization labels. We leverage wav2vec 2.0 to obtain superior and more nuanced results with the LB family audio stream. We use a bag-of-audio-words method with wav2vec 2.0 features to create high-level visualizations to understand family-infant vocalization interactions. We demonstrate that our high-quality visualizations capture major types of family vocalization interactions, in categories indicative of mental, behavioral, and developmental health, for both labeled and unlabeled LB audio.

  • Open Access

    Zoom, Zoom, Baby! Assessing Mother-Infant Interaction During the Still Face Paradigm and Infant Language Development via a Virtual Visit Procedure

    by Nancy L. McElwain, Yannan Hu, Xiaomei Li, Meghan C. Fisher, Jenny C. Baldwin, Jordan M. Bodway

    The COVID-19 pandemic has necessitated innovations in data collection protocols, including use of virtual or remote visits. Although developmental scientists used virtual visits prior to COVID-19, validation of virtual assessments of infant socioemotional and language development are lacking. We aimed to fill this gap by validating a virtual visit protocol that assesses mother and infant behavior during the Still Face Paradigm (SFP) and infant receptive and expressive communication using the Bayley-III Screening Test. Validation was accomplished through comparisons of data (i.e., proportions of missing data for a given task; observed infant and maternal behaviors) collected during in-person laboratory visits and virtual visits conducted via Zoom. Of the 119 mother-infant dyads who participated, 73 participated in lab visits only, 13 participated in virtual visits only, and 33 dyads participated in a combination of lab and virtual visits across four time points (3, 6, 9, and 12 months). Maternal perspectives of, and preferences for, virtual visits were also assessed. Proportions of missing data were higher during virtual visits, particularly for assessments of infant receptive communication. Nonetheless, comparisons of virtual and laboratory visits within a given time point (3, 6, or 9 months) indicated that mothers and infants showed similar proportions of facial expressions, vocalizations and directions of gaze during the SFP and infants showed similar and expected patterns of behavioral change across SFP episodes. Infants also demonstrated comparable expressive and receptive communicative abilities across virtual and laboratory assessments. Maternal reports of ease and preference for virtual visits varied by infant age, with mothers of 12-month-old infants reporting, on average, less ease of virtual visits and a preference for in-person visits. Results are discussed in terms of feasibility and validity of virtual visits for assessing infant socioemotional and language development, and broader advantages and disadvantages of virtual visits are also considered.