Katrin Baumgärtner

Nonlinear model predictive control (MPC) and reinforcement learning (RL) are two powerful control strategies with complementary advantages. This work shows how actor-critic RL techniques can be leveraged to improve the performance of MPC. The RL critic is used as an approximation of the optimal value function, and an actor rollout provides an initial guess for the primal variables of the MPC. A parallel control architecture is proposed where each MPC instance is solved twice for different initial guesses. Besides the actor rollout initialization, a shifted initialization from the previous solution is used. The control actions from the lowest-cost trajectory are applied to the system at each time step. We provide some theoretical justification of the proposed algorithm by establishing that the discounted closed-loop cost is upper-bounded by the discounted closed-loop cost of the original RL actor plus an error term that depends on the (sub)optimality of the RL actor and the accuracy of the critic. These results do not require globally optimal solutions and indicate that larger horizons mitigate the effect of errors in the critic approximation. The proposed algorithm is intended for applications where standard methods to construct terminal costs or constraints for MPC are impractical. The approach is demonstrated in an illustrative toy example and an autonomous driving overtaking scenario.

This study presents a novel state estimation approach integrating Deep Neural Networks (DNNs) into Moving Horizon Estimation (MHE). This is a shift from using traditional physics-based models within MHE towards data-driven techniques. Specifically, a Long Short-Term Memory (LSTM)-based DNN is trained using synthetic data derived from a high-fidelity thermal model of a Permanent Magnet Synchronous Machine (PMSM), applied within a thermal derating torque control strategy for battery electric vehicles. The trained DNN is directly embedded within an MHE formulation, forming a discrete-time nonlinear optimal control problem (OCP) solved via the acados optimization framework. Model-in-the-Loop simulations demonstrate accurate temperature estimation even under noisy sensor conditions and simulated sensor failures. Real-time implementation on embedded hardware confirms practical feasibility, achieving computational performance exceeding real-time requirements threefold. By integrating the learned LSTM-based dynamics directly into MHE, this work achieves state estimation accuracy, robustness, and adaptability while reducing modeling efforts and complexity. Overall, the results highlight the effectiveness of combining model-based and data-driven methods in safety-critical automotive control systems.

This study introduces a novel state estimation framework that incorporates Deep Neural Networks (DNNs) into Moving Horizon Estimation (MHE), shifting from traditional physics-based models to rapidly developed data-driven techniques. A DNN model with Long Short-Term Memory (LSTM) nodes is trained on synthetic data generated by a high-fidelity thermal model of a Permanent Magnet Synchronous Machine (PMSM), which undergoes thermal derating as part of the torque control strategy in a battery electric vehicle. The MHE is constructed by integrating the trained DNN with a simplified driving dynamics model in a discrete-time formulation, incorporating the LSTM hidden and cell states in the state vector to retain system dynamics. The resulting optimal control problem (OCP) is formulated as a nonlinear program (NLP) and implemented using the \texttt{acados} framework. Model-in-the-loop (MiL) simulations demonstrate accurate temperature estimation, even under noisy sensor conditions or failures. Achieving threefold real-time capability on embedded hardware confirms the feasibility of the approach for practical deployment. The primary focus of this study is to assess the feasibility of the MHE framework using a DNN-based plant model instead of focusing on quantitative comparisons of vehicle performance. Overall, this research highlights the potential of DNN-based MHE for real-time, safety-critical applications by combining the strengths of model-based and data-driven methods.

This paper considers Bayesian optimization (BO) for problems with known outer problem structure. In contrast to the classic BO setting, where the objective function itself is unknown and needs to be iteratively estimated from noisy observations, we analyze the case where the objective has a known outer structure - given in terms of a loss function - while the inner structure - an unknown input-output model - is again iteratively estimated from noisy observations of the model outputs. We introduce a novel lower confidence bound algorithm for this particular problem class which exploits the known outer problem structure. The proposed method is analyzed in terms of regret for the special case of convex loss functions and probabilistic parametric models which are linear in the uncertain parameters. Numerical examples illustrate the superior performance of structure-exploiting methods compared to structure-agnostic approaches.

The fields of MPC and RL consider two successful control techniques for Markov decision processes. Both approaches are derived from similar fundamental principles, and both are widely used in practical applications, including robotics, process control, energy systems, and autonomous driving. Despite their similarities, MPC and RL follow distinct paradigms that emerged from diverse communities and different requirements. Various technical discrepancies, particularly the role of an environment model as part of the algorithm, lead to methodologies with nearly complementary advantages. Due to their orthogonal benefits, research interest in combination methods has recently increased significantly, leading to a large and growing set of complex ideas leveraging MPC and RL. This work illuminates the differences, similarities, and fundamentals that allow for different combination algorithms and categorizes existing work accordingly. Particularly, we focus on the versatile actor-critic RL approach as a basis for our categorization and examine how the online optimization approach of MPC can be used to improve the overall closed-loop performance of a policy.

This paper examines the question of finding feasible points to discrete-time optimal control problems. The optimization problem of finding a feasible trajectory is transcribed to an unconstrained optimal control problem. An efficient algorithm, called FP-DDP, is proposed that solves the resulting problem using Differential Dynamic Programming preserving feasibility with respect to the system dynamics in every iteration. Notably, FP-DDP admits global and rapid local convergence properties induced by a combination of a Levenberg-Marquardt method and an Armijo-type line search. An efficient implementation of FP-DDP within acados is compared to established methods such as Direct Multiple Shooting, Direct Single Shooting, and state-of-the-art solvers.

We investigate the suboptimality resulting from the application of nominal model predictive control (MPC) to a nonlinear discrete time stochastic system. The suboptimality is defined with respect to the corresponding stochastic optimal control problem (OCP) that minimizes the expected cost of the closed loop system. In this context, nominal MPC corresponds to a form of certainty-equivalent control (CEC). We prove that, in a smooth and unconstrained setting, the suboptimality growth is of fourth order with respect to the level of uncertainty, a parameter which we can think of as a standard deviation. This implies that the suboptimality does not grow very quickly as the level of uncertainty is increased, providing further insight into the practical success of nominal MPC. Similarly, the difference between the optimal and suboptimal control inputs is of second order. We illustrate the result on a simple numerical example, which we also use to show how the proven relationship may cease to hold in the presence of state constraints.