JOJorge Ortiz

Jorge Ortiz

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Jorge Ortiz has a total of 32 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • VCHAR: A Variance-Driven Framework for Complex Human Activity Recognition With Vague Label

    by Sun Yuan, Salami Pargoo Navid, Ortiz Jorge

    Complex human activity recognition (CHAR) remains a pivotal challenge within ubiquitous computing, especially in the context of smart environments. Existing studies typically require meticulous labeling of both lower level and higher level activities—a task that is labor-intensive and prone to errors, focusing on datasets that precisely label simple activities or their sequences. These approaches are often impractical in real-world settings due to the scarcity and inaccuracies of available datasets. In response to these challenges, we introduce variance-driven complex human activity recognition (VCHAR), a novel framework capable of detecting both low-level activities and their corresponding high-level activities without requiring precise temporal or sequential labeling of the lower level activities. We introduce a novel approach by applying the KL divergence as the objective function, modeling the outputs of lower level activities as distributions over target time intervals while concurrently detecting higher level activities. This method only requires vaguely labeled activities, removing the need for precise, second-by-second annotations. Additionally, to enhance the layperson's understanding of our model's results, we are the first to apply a novel generative method that explains the result of the sensor model through video, making the information more accessible and understandable. Our evaluation across three publicly available datasets demonstrates that VCHAR uses KL divergence to model the lower level activity distribution over specific time intervals, enhancing the accuracy of complex activity recognition. Furthermore, user studies confirm that VCHAR's explanations are more intelligible compared to existing methods, facilitating a broader understanding of complex activity recognition among nonexperts.

  • Open Access

    Urban Sensing for Human-Centered Systems: A Modular Edge Framework for Real-Time Interaction

    by Navid Salami Pargoo, Mahshid Ghasemi, Shuren Xia, Mehmet Kerem Turkcan, Taqiya Ehsan, Chengbo Zang, Yuan Sun, Javad Ghaderi, Gil Zussman, Zoran Kostic, et al.

  • GeXSe (Generative Explanatory Sensor System): A Deep Generative Method for Human Activity Recognition of Smart Spaces IOT

    by Sun Yuan, Salami Pargoo Navid, Ortiz Jorge

    Effective sensemaking is essential across various domains, particularly in Human Activity Recognition (HAR), where trust in system outputs plays a critical role. A key challenge in HAR systems for smart spaces lies in their reliance on sensor-based activity recognition, which often lacks interpretability. This highlights the pressing need for models that not only achieve high accuracy in processing raw sensor data but also provide comprehensible insights, enabling laypersons to effectively utilize this otherwise invisible information in smart environments.Our work introduces GeXSe, a novel multi-task framework that jointly models raw sensors for classification while also generating grounded visual explanations. At the core of GeXSe lies a Parallel Multi-Branch Multi-Layer Perceptron Fast Fourier Convolution (PMB-MLP-FFC) module tailored for multi-modal sensor fusion and explanation generation. PMB-MLP-FFC extracts interpretable features optimized for both tasks through multi-branch parallel convolutions and Fourier transforms. We validated GeXSe across 3 diverse public datasets of daily activities recorded by microphone, motion, and environmental sensors. Results show superior activity recognition over baseline models. Furthermore, human evaluation studies confirm the generated visual explanations enhance understanding and trust compared to purely sensor-based outputs.


  • Open Access

    Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF

    by Yuan Sun, Navid Salami Pargoo, Peter Jin, Jorge Ortiz

    Reinforcement Learning from Human Feedback (RLHF) is popular in large language models (LLMs), whereas traditional Reinforcement Learning (RL) often falls short. Current autonomous driving methods typically utilize either human feedback in machine learning, including RL, or LLMs. Most feedback guides the car agent's learning process (e.g., controlling the car). RLHF is usually applied in the fine-tuning step, requiring direct human "preferences," which are not commonly used in optimizing autonomous driving models. In this research, we innovatively combine RLHF and LLMs to enhance autonomous driving safety. Training a model with human guidance from scratch is inefficient. Our framework starts with a pre-trained autonomous car agent model and implements multiple human-controlled agents, such as cars and pedestrians, to simulate real-life road environments. The autonomous car model is not directly controlled by humans. We integrate both physical and physiological feedback to fine-tune the model, optimizing this process using LLMs. This multi-agent interactive environment ensures safe, realistic interactions before real-world application. Finally, we will validate our model using data gathered from real-life testbeds located in New Jersey and New York City.


  • Open Access

    (Social) Trouble on the Road: Understanding and Addressing Social Discomfort in Shared Car Trips

    by Alexandra W.D. Bremers, Natalie Friedman, Sam Lee, Tong Wu, Eric Laurier, Malte F Jung, Jorge Ortiz, Wendy Ju

    Unpleasant social interactions on the road can negatively affect driving safety. At the same time, researchers have attempted to address social discomfort by exploring Conversational User Interfaces (CUIs) as social mediators. Before knowing whether CUIs could reduce social discomfort in a car, it is necessary to understand the nature of social discomfort in shared rides. To this end, we recorded nine families going on drives and performed interaction analysis on this data. We define three strategies to address social discomfort: contextual mediation, social mediation, and social support. We discuss considerations for engineering and design, and explore the limitations of current large language models in addressing social discomfort on the road.


  • Open Access

    Poster Abstract: A Radar Based User Discrimination System for Medication Adherence Monitoring | Proceedings of the 22nd International Conference on Information Processing in Sensor Networks

    by Murtadha Aldeer, David Waterworth, Parth Jain, Xiang Meng, Richard P. Martin, Jorge Ortiz

    Medication non-adherence is a major healthcare challenge globally, with over half of patients with chronic conditions in developed countries failing to follow their prescribed medication regimen. This can lead to poor disease outcomes, increased hospital visits, and a significant financial burden on healthcare systems [1]. These issues have driven a recent wave of research, including the development of smart adherence products [6] that can be incorporated into a patient’s daily life to monitor medication adherence. In this work, we present a radar-based system for user identification while taking medication, which extends our recent work [5]. we conducted preliminary experiments examining semi-medication-taking activities executed by 6 subjects. Our system achieved 80% accuracy in identifying who has taken the medication in a group of 3 subjects.

  • Open Access

    Poster Abstract: Multi-sensor Fusion for In-cabin Vehicular Sensing Applications | The 22nd International Conference on Information Processing in Sensor Networks

    by Tong Wu, Navid Salami Pargoo, Jorge Ortiz

    Cyber-physical-human systems (CPHS) in AI-based driver assistance applications require the integration of data from diverse modalities. These in-cabin CPHSs offer rich sensing capabilities, encompassing the vehicle, its surroundings, and the driver. One of the primary challenges in CPHSs is the incorporation of human behavior modeling to steer interactions that bolster human performance. This study introduces an in-cabin vehicular sensing framework that merges a driving simulator with a CPHS, thereby enabling researchers to devise and test multi-sensor fusion methodologies in human-in-the-loop driving situations. Initial experiments reveal that a driver interruptibility model can be effectively trained using the data gathered from our system.

  • Open Access

    Poster Abstract: A Testbed for Context Representation in Physical Spaces | The 22nd International Conference on Information Processing in Sensor Networks

    by Murtadha Aldeer, Yuan Sun, Nandana Pai, Joseph Florentine, Justin Yu, Jorge Ortiz

    The Internet of Things (IoT) offers transformative potential when combined with Machine Learning (ML), but labeling diverse IoT data remains challenging. To address this, we introduce SenseScape Testbed, an IoT experimentation platform for indoor environments with wireless sensor nodes, robots, and location-tracking nodes. This testbed enables IoT applications such as human activity recognition and indoor mobility tracking, supporting energy efficiency, occupant comfort, and context representation while providing a versatile environment for labeling and testing to advance ML algorithms tailored for IoT applications.


  • Open Access

    Learning When Agents Can Talk to Drivers Using the INAGT Dataset and Multisensor Fusion

    by Tong Wu, Nikolas Martelaro, Simon Stent, Jorge Ortiz, Wendy Ju

    This paper examines sensor fusion techniques for modeling opportunities for proactive speech-based in-car interfaces. We leverage the Is Now a Good Time (INAGT) dataset, which consists of automotive, physiological, and visual data collected from drivers who self-annotated responses to the question "Is now a good time?," indicating the opportunity to receive non-driving information during a 50-minute drive. We augment this original driver-annotated data with third-party annotations of perceived safety, in order to explore potential driver overconfidence. We show that fusing automotive, physiological, and visual data allows us to predict driver labels of availability, achieving an 0.874 F1-score by extracting statistically relevant features and training with our proposed deep neural network, PazNet. Using the same data and network, we achieve an 0.891 F1-score for predicting third-party labeled safe moments. We train these models to avoid false positives---determinations that it is a good time to interrupt when it is not---since false positives may cause driver distraction or service deactivation by the driver. Our analyses show that conservative models still leave many moments for interaction and show that most inopportune moments are short. This work lays a foundation for using sensor fusion models to predict when proactive speech systems should engage with drivers.