Jumana Abu-Khalaf

A fundamental limitation of modern conversational AI is its limited capacity to demonstrate sustained empathy in long-form interactions. We propose SCIRAG (Semantic Context Improvisational Retrieval-Augmented Generation), a feedback-driven retrieval framework for adaptive empathic dialogue. It employs a dual-loop retrieval framework, iteratively optimizing a static counseling dataset through user metadata and feedback memory refinement. To enhance contextual alignment, we deploy retrieval adaptation, enabling the model to retain and leverage past conversational cues based on user preferences. When integrated with Mixtral-8x7B, SCIRAG improves human-rated empathic understanding by +1.26 points and empathic response by +1.00 point on the RoPE scale, while increasing acceptability by +7.66 points compared to a fine-tuned non-RAG baseline. Automatic evaluation further shows gains in semantic alignment (BERTScore-F1 +0.11) and fluency (perplexity reduced from 19.1 to 12.3). We also present EMPATHIC , a dataset of unscripted, therapeutic conversations. Unlike conventional datasets that contain only 2-4 dialogue turns per conversation, EMPATHIC provides extended conversational trajectories (50+ turns per session), allowing models to learn long-range coherence and empathic listening. The proposed dataset will be publicly released.

Panoramic images play an important role in indoor robotics, offering a comprehensive spatial understanding of the environment. For tasks such as visual room rearrangement, where agents restore objects to their original positions or states, panoramic views provide a broader understanding of all changes. Existing 2D datasets for scene change understanding are limited to single-view images, which fail to capture the full spatial context, object relationships, and partially visible or obscured objects, making them less suitable for embodied AI applications. To address this, we introduce PanoSCU (Panoramic Scene Change Understanding), a dataset specifically designed to enhance the visual object rearrangement task. PanoSCU supports eight research tasks using single-viewand panoramic inputs: single-viewand panoramic detection, single-view and panoramic segmentation, single-view and panoramic change understanding, embodied object tracking, and change reversal. Our dataset contains 5,300 simulator-generated panoramas, encompassing 48 frequently-seen indoor object classes. We also introduce PanoStitch, a training-free stitching method for automatic panoramic data collection in embodied AI simulators. We evaluate state-of-the-art methods on panoramic segmentation and change understanding tasks. There is a gap in existing methods, as they are not designed for panoramic inputs and struggle with varying ratios and sizes, resulting from the unique challenges of visual object rearrangement. Our findings reveal these limitations and highlight PanoSCU’s challenging nature, emphasizing the need for further research to bridge this gap for panoramic indoor scene understanding. The dataset and the baseline methods for panoramic segmentation task are available publicly a.

A screen-printed wearable sensor accurately monitored respiratory rate in 15 participants with a 0.055 bpm error, showing promise for point-of-care use in conditions like COPD and COVID-19.

Embodied AI aims to develop robots that can \textit{understand} and execute human language instructions, as well as communicate in natural languages. On this front, we study the task of generating highly detailed navigational instructions for the embodied robots to follow. Although recent studies have demonstrated significant leaps in the generation of step-by-step instructions from sequences of images, the generated instructions lack variety in terms of their referral to objects and landmarks. Existing speaker models learn strategies to evade the evaluation metrics and obtain higher scores even for low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker), an instruction generator or \textit{Speaker} model that utilises both structural and semantic knowledge of the environment to produce richer instructions. For training, we employ a reward learning method in an adversarial setting to avoid systematic bias introduced by language evaluation metrics. Empirically, our method outperforms existing instruction generation models, evaluated using standard metrics. Our code is available at \url{https://github.com/gmuraleekrishna/SAS}.

Muraleekrishna Gopinathan, Martin Masek, Jumana Abu-Khalaf, David Suter. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.