Dayoung Jeong

Public surveys are indispensable resources for understanding social dynamics, yet their analysis often imposes a high cognitive load due to structural complexity. In this paper, we present LAPS, a Large Language Model (LLM)-assisted automated framework that supports end-to-end, hypothesis-driven statistical analysis of survey data. LAPS consists of four modules (i.e., Operationalization, Planning, Execution, and Reporting) with human-in-the-loop mechanisms to balance automation with user agency. To evaluate the applicability of LAPS, we conducted a within-subjects user study with 12 social science researchers across three analytical environments: traditional statistical tools, a general-purpose LLM, and LAPS. Our findings demonstrate that LAPS ensures researcher agency and analytical stability, reduces the cognitive burden in the analysis workflow, and produces trustworthy, coherent outputs. Based on these findings, we reflect on how LAPS improves researchers’ workflows and discuss design implications for scalable and trustworthy human-AI collaboration in survey-based research.

Cybersickness in virtual reality (VR) significantly disrupts user immersion. Although recent studies have proposed cybersickness prediction models, existing models have considered the moment of cybersickness onset, limiting their applicability in proactive detection. To address this limitation, we used long-term time series forecasting (LTSF) models based on multimodal sensor data collected from the head-mounted display (HMD). We used a pre-trained large language model (LLM) to effectively learn the salient features (e.g., seasonality) of multimodal sensor data by understanding the nuanced context within the data. The results of our experiment demonstrated that our model achieved comparable performance to the baseline models, with an MAE of 0.971 and an RMSE of 1.696. This indicates the potential for early prediction of cybersickness by employing LLM- and LTSF-based models with multimodal sensor data, suggesting a new direction in model development.

Cybersickness, a factor that hinders user immersion in VR, has been the subject of ongoing attempts to predict it using AI. Previous studies have used CNN and LSTM for prediction models and used attention mechanisms and XAI for data analysis, yet none explored a transformer that can better reflect the spatial and temporal characteristics of the data, beneficial for enhancing prediction and feature importance analysis. In this paper, we propose cybersickness prediction models using multimodal time-series sensor data (i.e., eye movement, head movement, and physiological signals) based on a transformer algorithm, considering sensor data pre-processing and multimodal data fusion methods. We constructed the MSCVR dataset consisting of normalized sensor data, spectrogram formatted sensor data, and cybersickness levels collected from 45 participants through a user study. We proposed two methods for embedding multimodal time-series sensor data into the transformer: modality-specific spatial and temporal transformer encoders for normalized sensor data (MS-STTN) and modality-specific spatial-temporal transformer encoder for spectrogram (MS-STTS). MS-STTN yielded the highest performance in the ablation study and the comparison of the existing models. Furthermore, by analyzing the importance of data features, we determined their relevance to cybersickness over time, especially the salience of eye movement features. Our results and insights derived from multimodal time-series sensor data and the transformer model provide a comprehensive understanding of cybersickness and its association with sensor data. Our MSCVR dataset and code are publicly available: https://github.com/dayoung-jeong/PRECYSE.git.

Maintaining a daily routine has profound implications for physical, emotional, and social well-being. Autistic individuals may experience various challenges in establishing and maintaining a healthy daily routine due to their tendency to be inactive in daily life combined with their characteristics and preferences. Previous studies employing mobile technology to support autistic individuals have primarily focused on self-help functions, with limited exploration into the detailed needs of these individuals to develop and maintain personalized routines. In this study, we conducted a nine-week field study with 18 autistic individuals using RoutineAid, a gamified app designed to support key routines of autistic individuals (i.e., physical activity, diet, mindfulness, and sleep). Our analysis incorporated five measures of self-evaluation on daily life, app usage logs, Fitbit physical activity data, and interviews. Our findings demonstrate the effectiveness of RoutineAid and highlight its two primary affordances for autistic individuals: (1) promoting self-efficacy and embedding health behavior and (2) refining daily routines for healthier outcomes. We discuss salient design insights for developing daily routine management systems for autistic individuals.

Background: Individuals with autism often experience heightened anxiety in workplace environments because of challenges in communication and sensory overload. As these experiences can result in negative self-image, promoting their self-efficacy in the workplace is crucial. Virtual reality (VR) systems have emerged as promising tools for enhancing the self-efficacy of individuals with autism in navigating social scenarios, aiding in the identification of anxiety-inducing situations, and preparing for real-world interactions. However, there is limited research exploring the potential of VR to enhance self-efficacy by facilitating an understanding of emotional and physiological states during social skills practice. Objective: This study aims to develop and evaluate a VR system that enabled users to experience simulated work-related social scenarios and reflect on their behavioral and physiological data through data visualizations. We intended to investigate how these data, combined with the simulations, can support individuals with autism in building their self-efficacy in social skills. Methods: We developed WorkplaceVR, a comprehensive VR system designed for engagement in simulated work-related social scenarios, supplemented with data-driven reflections of users’ behavioral and physiological responses. A within-subject deployment study was subsequently conducted with 14 young adults with autism to examine WorkplaceVR’s feasibility. A mixed methods approach was used, compassing pre- and postsystem use assessments of participants’ self-efficacy perceptions. Results: The study results revealed WorkplaceVR’s effectiveness in enhancing social skills and self-efficacy among individuals with autism. First, participants exhibited a statistically significant increase in perceived self-efficacy following their engagement with the VR system (P=.02). Second, thematic analysis of the interview data confirmed that the VR system and reflections on the data fostered increased self-awareness among participants about social situations that trigger their anxiety, as well as the behaviors they exhibit during anxious moments. This increased self-awareness prompted the participants to recollect their related experiences in the real world and articulate anxiety management strategies. Furthermore, the insights uncovered motivated participants to engage in self-advocacy, as they wanted to share the insights with others. Conclusions: This study highlights the potential of VR simulations enriched with physiological and behavioral sensing as a valuable tool for augmenting self-efficacy in workplace social interactions for individuals with autism. Data reflection facilitated by physiological sensors helped participants with autism become more self-aware of their emotions and behaviors, advocate for their characteristics, and develop positive self-beliefs.

Stress plays a significant role in fatal accidents, highlighting the importance of timely monitoring of driver stress to facilitate effective interventions and reduce road accidents. However, monitoring driver stress presents numerous challenges in the context of driving. First, state-of-the-art techniques such as self-stress evaluation and periodic cortisol level checks are not suitable for the driving scenario. Second, existing unimodal solutions does not provide a comprehensive and holistic assessment of the driver’s stress. Although some research utilizes multimodal features, the use of wearables attached to the driver’s body in real-life situations is impractical and highly discomforting. Our proposed solution tackles these challenges by offering a contactless and non-intrusive approach that prioritizes the driver’s comfort during the collection of multimodal data, which includes capturing heart rate variability (HRV), respiration rate, and microfacial expressions. Through feature-level data fusion, we combine and integrate these diverse modalities to generate comprehensive insights. These insights are then utilized by the multimodal learning pipeline to predict the driver’s stress levels in real driving scenarios.

BACKGROUND Individuals with autism often experience heightened anxiety in workplace environments because of challenges in communication and sensory overload. As these experiences can result in negative self-image, promoting their self-efficacy in the workplace is crucial. Virtual reality (VR) systems have emerged as promising tools for enhancing the self-efficacy of individuals with autism in navigating social scenarios, aiding in the identification of anxiety-inducing situations, and preparing for real-world interactions. However, there is limited research exploring the potential of VR to enhance self-efficacy by facilitating an understanding of emotional and physiological states during social skills practice. OBJECTIVE This study aims to develop and evaluate a VR system that enabled users to experience simulated work-related social scenarios and reflect on their behavioral and physiological data through data visualizations. We intended to investigate how these data, combined with the simulations, can support individuals with autism in building their self-efficacy in social skills. METHODS We developed WorkplaceVR, a comprehensive VR system designed for engagement in simulated work-related social scenarios, supplemented with data-driven reflections of users’ behavioral and physiological responses. A within-subject deployment study was subsequently conducted with 14 young adults with autism to examine WorkplaceVR’s feasibility. A mixed methods approach was used, compassing pre- and postsystem use assessments of participants’ self-efficacy perceptions. RESULTS The study results revealed WorkplaceVR’s effectiveness in enhancing social skills and self-efficacy among individuals with autism. First, participants exhibited a statistically significant increase in perceived self-efficacy following their engagement with the VR system (P=.02). Second, thematic analysis of the interview data confirmed that the VR system and reflections on the data fostered increased self-awareness among participants about social situations that trigger their anxiety, as well as the behaviors they exhibit during anxious moments. This increased self-awareness prompted the participants to recollect their related experiences in the real world and articulate anxiety management strategies. Furthermore, the insights uncovered motivated participants to engage in self-advocacy, as they wanted to share the insights with others. CONCLUSIONS This study highlights the potential of VR simulations enriched with physiological and behavioral sensing as a valuable tool for augmenting self-efficacy in workplace social interactions for individuals with autism. Data reflection facilitated by physiological sensors helped participants with autism become more self-aware of their emotions and behaviors, advocate for their characteristics, and develop positive self-beliefs.