Vinod Namboodiri

Smart assistants increasingly act proactively, yet mistimed or intrusive behavior often causes users to lose trust and disable these features. Learning user preferences for proactive assistance is difficult because real-world studies are costly, limited in scale, and rarely capture how preferences change across multiple interaction sessions. Large language model-based generative agents offer a way to simulate realistic interactions, but existing synthetic datasets remain limited in temporal depth, diverse personas, and multi-dimensional preferences. They also provide little support for transferring population-level insights to individual users under on-device constraints. We present a population-to-individual learning framework for preference-aligned proactive assistants that operates under on-device and privacy constraints. Our approach uses large-scale interaction simulation with 1,000 diverse personas to learn shared structure in how users express preferences across recurring dimensions such as timing, autonomy, and communication style, providing a strong cold start without relying on real user logs. The assistant then adapts to individual users on device through lightweight activation-based steering driven by simple interaction feedback, without model retraining or cloud-side updates. We evaluate the framework using controlled simulations with 1,000 simulated personas and a human-subject study with 34 participants. Results show improved timing decisions and perceived interaction quality over untuned and direct-response baselines, while on-device activation steering achieves performance comparable to reinforcement learning from human feedback. Participants also report higher satisfaction, trust, and comfort as the assistant adapts over multiple interaction sessions.

Indoor navigation poses significant challenges due to the ineffectiveness of satellite signals, which fail to penetrate building structures. This challenge is manifold greater for visually impaired persons (VIPs) who cannot obtain information from wayfinding signage. Other sensor signals (e.g., Bluetooth and WiFi) come into play and can be used to estimate users’ positions. However, these solutions typically require the addition of tags or hardware throughout the environment to calibrate Points of Interest (PoIs), which limits scalability in providing precise navigational guidance to VIPs. Thus, we propose a computer vision (CV)-based approach called NaVIP (Navigation for VIPs) that offers a cost-effective, infrastructure-free, and scalable solution to elaborate indoor navigation for VIPs. In addition to turn-by-turn navigation, NaVIP provides continuous and accurate distance and clockwise orientation cues, guiding users through any deviations or missteps from the starting point to their final destination. To confirm our conclusion, we extensively evaluated the localization accuracy of NaVIP within a large and complex indoor space. Furthermore, we demonstrate the potential effectiveness and usability of NaVIP by creating an iOS application and testing it with VIP users. NaVIP, as a CV-based framework, is designed to be task-scalable, enabling future enhancements that utilize the advancements in CV tasks, such as automated object detection and depth estimation, to aid VIPs further.

Smart assistants increasingly act proactively, yet mistimed or intrusive behavior often causes users to lose trust and disable these features. Learning user preferences for proactive assistance is difficult because real-world studies are costly, limited in scale, and rarely capture how preferences change across multiple interaction sessions. Large language model based generative agents offer a way to simulate realistic interactions, but existing synthetic datasets remain limited in temporal depth, diverse personas, and multi-dimensional preferences. They also provide little support for transferring population-level insights to individual users under on-device constraints. We present a population-to-individual learning framework for preference-aligned proactive assistants that operates under on-device and privacy constraints. Our approach uses large-scale interaction simulation with 1,000 diverse personas to learn shared structure in how users express preferences across recurring dimensions such as timing, autonomy, and communication style, providing a strong cold start without relying on real user logs. The assistant then adapts to individual users on device through lightweight activation-based steering driven by simple interaction feedback, without model retraining or cloud-side updates. We evaluate the framework using controlled simulations with 1,000 simulated personas and a human-subject study with 30 participants. Results show improved timing decisions and perceived interaction quality over untuned and direct-response baselines, while on-device activation steering achieves performance comparable to reinforcement learning from human feedback. Participants also report higher satisfaction, trust, and comfort as the assistant adapts over multiple sessions of interactions.

Designing and evaluating personalized and proactive assistant agents remains challenging due to the time, cost, and ethical concerns associated with human-in-the-loop experimentation. Existing Human-Computer Interaction (HCI) methods often require extensive physical setup and human participation, which introduces privacy concerns and limits scalability. Simulated environments offer a partial solution but are typically constrained by rule-based scenarios and still depend heavily on human input to guide interactions and interpret results. Recent advances in large language models (LLMs) have introduced the possibility of generative agents that can simulate realistic human behavior, reasoning, and social dynamics. However, their effectiveness in modeling human-assistant interactions remains largely unexplored. To address this gap, we present a generative agent-based simulation platform designed to simulate human-assistant interactions. We identify ten prior studies on assistant agents that span different aspects of interaction design and replicate these studies using our simulation platform. Our results show that fully simulated experiments using generative agents can approximate key aspects of human-assistant interactions. Based on these simulations, we are able to replicate the core conclusions of the original studies. Our work provides a scalable and cost-effective approach for studying assistant agent design without requiring live human subjects. Additional resources and project materials are available at https://dash-gidea.github.io/.

Human Activity Recognition (HAR) using Inertial Measurement Unit (IMU) sensors is critical for applications in healthcare, safety, and industrial production. However, variations in activity patterns, device types, and sensor placements create distribution gaps across datasets, reducing the performance of HAR models. To address this, we propose LanHAR, a novel system that leverages Large Language Models (LLMs) to generate semantic interpretations of sensor readings and activity labels for cross-dataset HAR. This approach not only mitigates cross-dataset heterogeneity but also enhances the recognition of new activities. LanHAR employs an iterative re-generation method to produce high-quality semantic interpretations with LLMs and a two-stage training framework that bridges the semantic interpretations of sensor readings and activity labels. This ultimately leads to a lightweight sensor encoder suitable for mobile deployment, enabling any sensor reading to be mapped into the semantic interpretation space. Experiments on five public datasets demonstrate that our approach significantly outperforms state-of-the-art methods in both cross-dataset HAR and new activity recognition. The source code is publicly available at https://github.com/DASHLab/LanHAR.

Navigating unfamiliar indoor spaces presents significant challenges for blind or visually impaired (BVI) individuals due to complex layouts and limited accessible signage. This paper introduces AccessQuotient, a predictive metric for quantifying navigation difficulty in built environments. The model captures architectural complexity through decision and branching points, providing a lightweight and broadly applicable framework. Validation was conducted with participants with and without orientation and mobility (O\&M) training in a multi-level academic building, supplemented with comparative analysis across additional sites to illustrate sensitivity to architectural variation. By bridging probabilistic modeling with empirical observations, AccessQuotient offers a structured way to assess accessibility challenges before interventions are deployed. The framework establishes a foundation for extensions that incorporate environmental factors, personalization, and qualitative feedback from BVI users. Positioned alongside predictive approaches in digital accessibility, AccessQuotient highlights the potential of model-based metrics to guide inclusive design and inform accessibility standards in physical environments.

Emerging trends in technology are providing opportunities for a broader range of accessible and assistive technologies (AATs) to positively impact persons with disabilities in terms of independent living and employment within and across built environments. However, such technologies typically require significant investments by entities that offer such options. It is not clear how such firms compete in a market with other firms that may not provide such options. Understanding such competition can help to promote greater investments in accessibility infrastructure within built environments by entities and provide insights into how federal efforts can further boost such efforts. To this end, this paper presents a game-theoretic framework of market competition between two firms where one invests in accessibility (bearing additional upfront costs) and compares it with another one that does not. Numerical evaluations demonstrate the range of parametric values where accessibility investments pay off. Furthermore, case studies are presented to demonstrate the practical feasibility of these parameter values. The results indicate that any firm considering making accessibility investments can expect to make profits and also gain an advantage over its competitors if the expected increase in the average user experience is significant (quantified as 20% or more for the parameters considered in this work) across all potential users.