Wenhao Ding

The advent of end-to-end autonomy stacks—often lacking interpretable intermediate modules—has placed an increased burden on ensuring that the final output, i.e., the motion plan, is safe in order to validate the safety of the entire stack. This requires a safety monitor that is both complete (able to detect all unsafe plans) and sound (does not flag safe plans). In this work, we propose a principled safety monitor that leverages modern multi-modal trajectory predictors to approximate forward reachable sets (FRS) of surrounding agents. By formulating a convex program, we efficiently extract these data-driven FRSs directly from the predicted state distributions, conditioned on scene context such as lane topology and agent history. To ensure completeness, we leverage conformal prediction to calibrate the FRS and guarantee coverage of ground-truth trajectories with high probability. To preserve soundness in out-of-distribution (OOD) scenarios or under predictor failure, we introduce a Bayesian filter that dynamically adjusts the FRS conservativeness based on the predictor’s observed performance. We then assess the safety of the ego vehicle’s motion plan by checking for intersections with these calibrated FRSs, ensuring the plan remains collision-free under plausible future behaviors of others. Extensive experiments on the nuScenes dataset show our approach significantly improves soundness while maintaining completeness, offering a practical and reliable safety monitor for learned autonomy stacks.

Learning-based planners generate natural human-like driving behaviors by learning to reason about nuanced interactions from data, overcoming the rigid behaviors that arise from rule-based planners. Nonetheless, data-driven approaches often struggle with rare, safety-critical scenarios and offer limited controllability over the generated trajectories. To address these challenges, we propose RealDrive, a Retrieval-Augmented Generation (RAG) framework that initializes a diffusion-based planning policy by retrieving the most relevant expert demonstrations from the training dataset. By interpolating between current observations and retrieved examples through a denoising process, our approach enables fine-grained control and safe behavior across diverse scenarios, leveraging the strong prior provided by the retrieved scenario. Another key insight we produce is that a task-relevant retrieval model trained with planning-based objectives results in superior planning performance in our framework compared to a task-agnostic retriever. Experimental results demonstrate improved generalization to long-tail events and enhanced trajectory diversity compared to standard learning-based planners -- we observe a 40% reduction in collision rate on the Waymo Open Motion dataset with RAG.

Generating adversarial scenes that potentially fail autonomous driving systems provides an effective way to improve their robustness. Extending purely data-driven generative models, recent specialized models satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns implicitly at the neuron level. In this paper, we introduce a method to incorporate domain knowledge explicitly in the generation process to achieve Semantically Adversarial Generation (SAG) . To be consistent with the composition of driving scenes, we first categorize the knowledge into two types, the property of objects and the relationship among objects. We then propose a tree-structured variational auto-encoder (T-VAE) to learn hierarchical scene representation. By imposing semantic rules on the properties of nodes and edges into the tree structure, explicit knowledge integration enables controllable generation. To demonstrate the advantage of structural representation, we construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting. We further extend to realistic environments for autonomous vehicles, showing that our method efficiently identifies adversarial driving scenes against different state-of-the-art 3D point cloud segmentation models and satisfies the constraints specified as explicit knowledge.

Detecting gait abnormalities is crucial for assessing fall risks and early identification of neuromusculoskeletal disorders such as Parkinson's and stroke. Traditional assessments in gait clinics are infrequent and pose barriers, particularly for disadvantaged populations. Previous efforts have explored sensor-based approaches for in-home gait assessments, yet they face limitations such as visual obstructions (cameras), limited coverage (pressure mats), and the need for device carrying (wearables and insoles). To overcome these limitations, we introduce an in-home gait abnormality detection system using footstep-induced floor vibrations, enabling low-cost, non-intrusive, device-free gait health monitoring. The main research challenge is the high uncertainty in floor vibrations due to gait variations among people, making it challenging to develop a generalizable model for new patients. To address this, we analyze time-frequency-domain features of floor vibration data during specific gait phases and develop a feature transformation method through contrastive learning to address the between-people gait variation challenge. Our method transforms the features from vibrations to an embedding space where samples from different people stay close to each other (robust to people variation) while normal and abnormal gait samples are far apart (sensitive to gait abnormality). After that, gait abnormalities are detected by a downstream classifier after feature transformation. We evaluated our approach through a real-world walking experiment with 21 participants and achieved an 85% to 95% mean accuracy in detecting various gait abnormalities. This novel method overcomes prior limitations in in-home gait assessments, offering accessible gait abnormality detection without the need for intrusive devices or labels for new patients.

In the domain of autonomous driving, the offline Reinforcement Learning (RL) approaches exhibit notable efficacy in addressing sequential decision-making problems from offline datasets. However, maintaining safety in diverse safety-critical scenarios remains a significant challenge due to long-tailed and unforeseen scenarios absent from offline datasets. In this letter, we introduce the saFety-aware strUctured Scenario representatION (FUSION), a pioneering representation learning method in offline RL to facilitate the learning of a generalizable end-to-end driving policy by leveraging structured scenario information. FUSION capitalizes on the causal relationships between the decomposed reward, cost, state, and action space, constructing a framework for structured sequential reasoning in dynamic traffic environments. We conduct extensive evaluations in two typical real-world settings of the distribution shift in autonomous vehicles, demonstrating the good balance between safety cost and utility reward compared to the current state-of-the-art safe RL and IL baselines. Empirical evidence in various driving scenarios attests that FUSION significantly enhances the safety and generalizability of autonomous driving agents, even in the face of challenging and unseen environments. Furthermore, our ablation studies reveal noticeable improvements in the integration of causal representation into the offline safe RL algorithm. Our code implementation is available on the project website.

Crowd monitoring involves tracking and analyzing the behavior of large groups of people in large-scale public spaces, such as sports games. In sports stadiums, understanding audience reactions to the games and their distribution around the public facilities is important for ensuring public safety and security, enhancing the game experience, and improving crowd management. Recent crowd-crushing incidents (e.g., Kanjuruhan Stadium disaster, Seoul Halloween Stampede) have caused 100+ deaths in a single event, calling for advancements in crowd monitoring methods. Existing monitoring approaches include manual observation, wearables, video-, audio-, and WiFi-based sensing. However, few meet the practical needs due to their limitations in cost, privacy protection, and accuracy. In this paper, we introduce GameVibes, a novel method for crowd behavior monitoring using crowd-induced floor vibrations to infer audience reactions to the game (e.g., clapping, stomping, dancing) and crowd traffic (i.e., the number of people entering each door). The main benefits of GameVibes are that it allows continuous, fine-grained crowd monitoring in a cost-effective and non-intrusive way and is perceived as more privacy-friendly. Unlike monitoring an individual person, crowd monitoring involves understanding the overall behavior of a large population (typically more than 1,000), leading to high uncertainty in the vibration data. To overcome the challenge, we first establish the game and facility association to inform the context of crowd behaviors, including 1) game associations (temporal context) between the crowd reaction and the game progress and 2) facility associations (spatial context) between the crowd traffic and facility layouts. Then, we formulate the crowd monitoring problem by converting the conceptual graph of the audience-game-facility association into probabilistic game/facility association models. Through these models, GameVibes first learns the latent representations of the game progress and facility layout through neural network encoders, and then integrates heterogeneous game/facility information and vibration data to estimate crowd behaviors. This mitigates the estimation error due to the uncertainty in vibration data. To evaluate our approach, we conduct 6 real-world deployments for NCAA Pac-12 games at Stanford Maples Pavilion. Our results show that GameVibes achieves a 0.9 F-1 score in crowd reaction monitoring and 9.3 mean absolute error in crowd traffic estimation, which correspond to 10% and 12.2% error reduction, respectively, compared to the baseline methods without context-specific information.

Robustness has been extensively studied in reinforcement learning (RL) to handle various forms of uncertainty such as random perturbations, rare events, and malicious attacks. In this work, we consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. These spurious correlations are ubiquitous in real-world tasks, for instance, a self-driving car usually observes heavy traffic in the daytime and light traffic at night due to unobservable human activity. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Although motivated, enabling robustness against spurious correlation poses significant challenges since the uncertainty set, shaped by the unobserved confounder and causal structure, is difficult to characterize and identify. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge. To solve this issue, we propose Robust State-Confounded Markov Decision Processes (RSC-MDPs) and theoretically demonstrate its superiority in avoiding learning spurious correlations compared with other robust RL counterparts. We also design an empirical algorithm to learn the robust optimal policy for RSC-MDPs, which outperforms all baselines in eight realistic self-driving and manipulation tasks.

Training control policies in simulation is more appealing than on real robots directly, as it allows for exploring diverse states in an efficient manner. Yet, robot simulators inevitably exhibit disparities from the real-world \rebut{dynamics}, yielding inaccuracies that manifest as the dynamical simulation-to-reality (sim-to-real) gap. Existing literature has proposed to close this gap by actively modifying specific simulator parameters to align the simulated data with real-world observations. However, the set of tunable parameters is usually manually selected to reduce the search space in a case-by-case manner, which is hard to scale up for complex systems and requires extensive domain knowledge. To address the scalability issue and automate the parameter-tuning process, we introduce COMPASS, which aligns the simulator with the real world by discovering the causal relationship between the environment parameters and the sim-to-real gap. Concretely, our method learns a differentiable mapping from the environment parameters to the differences between simulated and real-world robot-object trajectories. This mapping is governed by a simultaneously learned causal graph to help prune the search space of parameters, provide better interpretability, and improve generalization on unseen parameters. We perform experiments to achieve both sim-to-sim and sim-to-real transfer, and show that our method has significant improvements in trajectory alignment and task success rate over strong baselines in several challenging manipulation tasks.