Paul Taele

Blind and visually impaired (BVI) students face significant challenges in traditional educational settings. While screen readers and braille materials offer some accessibility, they often lack interactivity and real-time adaptability to individual learning needs. This paper presents Audemy, an AI-powered audio-based learning platform designed to provide personalized, accessible, and engaging educational experiences for BVI students. Audemy uses adaptive learning techniques to customize content based on student accuracy, pacing preferences, and engagement patterns. The platform has been iteratively developed with input from over 20 educators specializing in accessibility and currently serves over 2,000 BVI students. Educator insights show key considerations for accessible AI, including the importance of engagement, intuitive design, compatibility with existing assistive technologies, and the role of positive reinforcement in maintaining student motivation. Beyond accessibility, this paper explores the ethical implications of AI in education, emphasizing data privacy, security, and transparency. Audemy demonstrates how AI can empower BVI students with personalized and equitable learning opportunities, advancing the broader goal of inclusive education.

Virtual reality (VR) can provide transformative educational experiences for individuals with intellectual disabilities (ID). To leverage this aspect in higher education, a team of faculty members and a transitional university program collaborated to integrate VR into their support systems. The initial VR experience sessions led to the development of specialized VR applications designed to support individuals with ID and raise awareness about their needs in the university setting. We detail a co-design process resulting in two VR projects: one for enhancing job interview skills and another for building empathy by simulating challenges faced by individuals with ID. This program highlights the effectiveness of inclusive participatory design in VR development, demonstrating how cross-ability collaboration leads to more accessible and relevant technological solutions. Our findings underscore VR's potential for skill development and fostering empathy, while contributing to the broader discourse on inclusive technology development and promoting a more inclusive society.

This study introduces an innovative approach to designing an audio-tactile interface for representing coordinate plane graphs. Traditional graphing methods heavily rely on visual elements, posing accessibility challenges for blind and visually impaired individuals in understanding this spatial concept. To identify challenges faced by blind students, we conducted a pre-design study with blind individuals to understand current challenges with spatial mathematics and current assistive technologies. Insights from this study informed the development of our system. Based on the insights obtained, we implemented the audio-based system. Using the Latin square method, a user study was conducted to compare different auditory cues (e.g., pitch representation, word-based description) and evaluate the integration of a tactile interface. Four distinct system variations were created for systematic evaluation, measuring mental/physical workload, enjoyability, and system intuitiveness. This research significantly contributes to the advancement of inclusive and intuitive multi-modal accessible math tools, enhancing the accessibility of learning experiences.

Monitoring activities of daily living (ADLs) plays an important role in measuring and responding to a person’s ability to manage their basic physical needs. Effective recognition systems for monitoring ADLs must successfully recognize naturalistic activities that also realistically occur at infrequent intervals. However, existing systems primarily focus on either recognizing more separable, controlled activity types or are trained on balanced datasets where activities occur more frequently. In our work, we investigate the challenges associated with applying machine learning to an imbalanced dataset collected from a fully in-the-wild environment. This analysis shows that the combination of preprocessing techniques to increase recall and postprocessing techniques to increase precision can result in more desirable models for tasks such as ADL monitoring. In a user-independent evaluation using in-the-wild data, these techniques resulted in a model that achieved an event-based F1-score of over 0.9 for brushing teeth, combing hair, walking, and washing hands. This work tackles fundamental challenges in machine learning that will need to be addressed in order for these systems to be deployed and reliably work in the real world.

The past several decades have seen significant advances in monitoring older adults' health and well-being. However, creating viable, practical monitoring systems for informing caregivers requires understanding which behaviors and signs to track and what approaches best present that information. To investigate how technology can be leveraged to better augment caregivers' workflows, we take a multi-stage, qualitative approach to gain insights into the needs of caregivers and the older adults receiving care. Specifically, we use a series of domain expert interviews, cognitive walkthroughs, and semi-structured interviews with residents, and we synthesize our takeaways using thematic analysis at each phase. Our results show that this type of monitoring technology has great potential to reduce the effort needed by caregivers to complete their responsibilities and communicate with their teams. Additionally, we found that older adults are receptive to the technology but their privacy and autonomy must be prioritized for the sake of their mental wellbeing. These insights will facilitate greater intelligent interface development for Person-Centered Care by identifying important design considerations and vital features that require system support.

This study introduces an innovative approach to design an interactive audio-based platform for the popular word game, Wordle. Unlike conventional word games that heavily rely on visual elements, there is a lack of alternatives in audio-based formats, leading to limited accessibility and engagement for a broader audience. To address this gap, we propose an audio-based game that transforms the traditional Wordle interface into a conversational format, exploring the possibilities of purely audio-based interactions. To justify design decisions for the proposed interface, we conducted a design study focusing on interaction cues between two players engaged in the Wordle game through conversation. Based on the insights obtained, we implemented the audio-based system. Subsequently, a user study was conducted to compare different input types (e.g., letter-based partitions, entire words) and evaluate the integration of an interactive question-based interface. Four distinct game variations were created for systematic evaluation, measuring mental/physical workload, enjoyability, and system intuitiveness. This research contributes to the development of inclusive and engaging audio-based word games, while providing insights into the benefits and challenges of incorporating interactive interfaces. It paves the way for a more accessible gaming experience and promotes the adoption of audio-based gaming interfaces in the broader gaming community.

BVI (blind and visually impaired) persons do not have the same degree of access to concepts and procedures in learning algebraic equations as sighted persons. As tactile resources are uncommon, most rely on auditory-based methods that place heavy stress on working memory. We propose a novel approach that leverages haptic technology to communicate mathematical symbols via vibrotactile patterns; this design addresses open challenges found in existing resources for BVI people and explores the potential utility of this medium for all users. We conducted three separate studies to evaluate the feasibility of our gloves that included BVI participants. The outcomes of our studies yielded insightful design discussions, including a blind participant who found the device immediately intuitive and exciting in potential.

The prevalence of virtual meeting software and webcams have normalized holding group meetings both remotely and distributed. However, virtual meetings reduce viewing to computing displays and listening to unidirectional speakers, which disrupt real-life social cues for activities such as informally polling participants “on-the-fly”. This paper proposes an intelligent virtual meetings app called Show of Hands, which leverages nonverbal communication cues to spontaneously poll virtual participants. The app recognizes virtual participants’ intuitive real-time hand gestures to express intended polling selections, and then displays in real-time highly-visible video filters that overlay participants’ camera views and a visual chart of the aggregated polling counts. Our work benefits virtual participants in seamlessly conducting spontaneous polls to gauge opinions or check knowledge of attendees without prior preparation, expressing poll responses using familiar physical gestures, and being better informed of poll results with both a distributed view of video filters and a focused chart visualization.