Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Johannes Ultsch has a total of 14 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    VDCWorkbench: A Source-Available Modelica Toolbox for Research and Education on Vehicle Dynamics and Control

    by Jonathan Brembeck, Ricardo de Castro, Johannes Rein, Iman Ebrahimi, Johannes Ultsch, Christoph Winter, Kenan Ahmic

  • Open Access

    train_fmu_gym: A Functional Mock-up Unit-Based Framework to Train Reinforcement Learning Agents for Multi-Physical Systems

    by Johannes Ultsch, Kenan Ahmic, Jonathan Brembeck

    Deep reinforcement learning (DRL) has become increasingly popular for solving sequential decision-making problems in control applications, particularly for multi-physical systems such as vehicle and robotic motion control. The application of DRL to control applications is especially appealing, as control engineering is strongly conducted with the utilization of simulation models. Leveraging these preexisting simulation models for DRL agent training provides a promising and effective strategy. Although DRL has gained popularity in controlling complex multi-physical systems, existing DRL frameworks still lack support for the entire training process of DRL agents utilizing preexisting simulation models. In this work, we present a structured workflow accompanied by its implementation, which utilizes the Functional Mock-up Interface (FMI) to leverage the use of preexisting simulation models for DRL training. We introduce the concept of scenarios as a set of environment parameters and formalize the concept of evaluation metrics. In addition, we extended the scenario concept to uncertain scenarios, to account for environmental uncertainties during training. Our proposed workflow is demonstrated on a path following control problem in automated driving, where a residual policy is learned to enhance the performance of a baseline controller. Using our workflow and toolchain implementation, we successfully trained an agent that achieved significant reductions in tracking errors, with maximum lateral displacement errors reduced by up to 93.4% and velocity tracking errors decreased by up to 41.3%. Our results demonstrates the effectiveness of our approach in leveraging preexisting simulation models for DRL training, paving the way for more efficient and effective control applications.


  • Predictive Safeguarding and Performance Monitoring for Learning-Based Vehicle Motion Controllers

    by Kenan Ahmic, Johannes Ultsch, Jonathan Brembeck, Darius Burschka

    A major issue with learning-based control methods is their lack of generalization in unseen parts of the state space. They tend to exhibit inefficient control performances or even complete failure in situations that were not encountered during the training process. This limitation is highly problematic in safety-critical applications, such as autonomous driving, where reliability and safety are essential. To address this issue, we propose a prediction-based safeguarding and performance monitoring framework for learning-based vehicle motion controllers. The proposed framework ensures a fail-safe execution during runtime and additionally intervenes if the predicted performance of the controller is safe but inefficient due to poor generalization capabilities. More specifically, in our framework, the behavior of the learning-based controller is predicted for a finite time horizon. If predictions are unsafe, a fallback controller is activated that ensures vehicle safety. If predictions are safe, the fallback controller's behavior is also predicted and compared to that of the learning-based controller. Afterwards, the more efficient controller is applied to the vehicle, which prevents the learning-based controller from executing safe but inefficient control behaviors. Simulation results for path following control applications demonstrate that the proposed framework is able to ensure vehicle safety at all times and that it is more efficient compared to the sole use of the fallback controller.

  • Open Access

    Reinforcement Learning for Semi-Active Vertical Dynamics Control with Real-World Tests

    by Johannes Ultsch, Andreas Pfeiffer, Julian Ruggaber, Tobias Kamp, Jonathan Brembeck, Jakub Tobolář

    In vertical vehicle dynamics control, semi-active dampers are used to enhance ride comfort and road-holding with only minor additional energy expenses. However, a complex control problem arises from the combined effects of (1) the constrained semi-active damper characteristic, (2) the opposing control objectives of improving ride comfort and road-holding, and (3) the additionally coupled vertical dynamic system. This work presents the application of Reinforcement Learning to the vertical dynamics control problem of a real street vehicle to address these issues. We discuss the entire Reinforcement Learning-based controller design process, which started with deriving a sufficiently accurate training model representing the vehicle behavior. The obtained model was then used to train a Reinforcement Learning agent, which offered improved vehicle ride qualities. After that, we verified the trained agent in a full-vehicle simulation setup before the agent was deployed in the real vehicle. Quantitative and qualitative real-world tests highlight the increased performance of the trained agent in comparison to a benchmark controller. Tests on a real-world four-post test rig showed that the trained RL-based controller was able to outperform an offline-optimized benchmark controller on road-like excitations, improving the comfort criterion by about 2.5% and the road-holding criterion by about 2.0% on average.


  • Open Access

    Reinforcement Learning-Based Path Following Control with Dynamics Randomization for Parametric Uncertainties in Autonomous Driving

    by Kenan Ahmic, Johannes Ultsch, Jonathan Brembeck, Christoph Winter

    Reinforcement learning-based controllers for safety-critical applications, such as autonomous driving, are typically trained in simulation, where a vehicle model is provided during the learning process. However, an inaccurate parameterization of the vehicle model used for training heavily influences the performance of the reinforcement learning agent during execution. This inaccuracy is either caused by changes due to environmental influences or by falsely estimated vehicle parameters. In this work, we present our approach of combining dynamics randomization with reinforcement learning to overcome this issue for a path-following control task of an autonomous and over-actuated robotic vehicle. We train three independent agents, where each agent experiences randomization for a different vehicle dynamics parameter, i.e., the mass, the yaw inertia, and the road-tire friction. We randomize the parameters uniformly within predefined ranges to enable the agents to learn an equally robust control behavior for all possible parameter values. Finally, in a simulation study, we compare the performance of the agents trained with dynamics randomization to the performance of an agent trained with the nominal parameter values. Simulation results demonstrate that the former agents obtain a higher level of robustness against model uncertainties and varying environmental conditions than the latter agent trained with nominal vehicle parameter values.


  • Open Access

    Advanced Controller Development Based on eFMI with Applications to Automotive Vertical Dynamics Control

    by Johannes Ultsch, Julian Ruggaber, Andreas Pfeiffer, Christina Schreppel, Jakub Tobolář, Jonathan Brembeck, Daniel Baumgartner

    High-level modeling languages facilitate system modeling and the development of control systems. This is mainly achieved by the automated handling of differential algebraic equations which describe the dynamics of the modeled systems across different physical domains. A wide selection of model libraries provides additional support to the modeling process. Nevertheless, deployment on embedded targets poses a challenge and usually requires manual modification and reimplementation of the control system. The novel proposed eFMI Standard (Functional Mock-up Interface for embedded systems) introduces a workflow and an automated toolchain to simplify the deployment of model-based control systems on embedded targets. This contribution describes the application and verification of the eFMI workflow using a vertical dynamics control problem with an automotive application as an example. The workflow is exemplified by a control system design process which is supported by the a-causal, multi-physical, high-level modeling language Modelica. In this process, the eFMI toolchain is applied to a model-based controller for semi-active dampers and demonstrated using an eFMI-based nonlinear prediction model within a nonlinear Kalman filter. The generated code was successfully tested in different validation steps on the dedicated embedded system. Additionally, tests with a low-volume production electronic control unit (ECU) in a series-produced car demonstrated the correct execution of the controller code under real-world conditions. The novelty of our approach is that it automatically derives an embedded software solution from a high-level multi-physical model with standardized eFMI methodology and tooling. We present one of the first full application scenarios (covering all aspects ranging from multi-physical modeling up to embedded target deployment) of the new eFMI tooling.


  • Open Access

    Learning-Based Cooperative Adaptive Cruise Control

    by Jonas Mirwald, Johannes Ultsch, Ricardo de Castro, Jonathan Brembeck

    Traffic congestion and the occurrence of traffic accidents are problems that can be mitigated by applying cooperative adaptive cruise control (CACC). In this work, we used deep reinforcement learning for CACC and assessed its potential to outperform model-based methods. The trade-off between distance-error minimization and energy consumption minimization whilst still ensuring operational safety was investigated. Alongside a string stability condition, robustness against burst errors in communication also was incorporated, and the effect of preview information was assessed. The controllers were trained using the proximal policy optimization algorithm. A validation by comparison with a model-based controller was performed. The performance of the trained controllers was verified with respect to the mean energy consumption and the root mean squared distance error. In our evaluation scenarios, the learning-based controllers reduced energy consumption in comparison to the model-based controller by 17.9% on average.