DiAReL: Reinforcement Learning With Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control
Open Access
- 1 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Control Systems Technology
- Vol. 34 (2) , 1037-1043
Abstract
Delayed Markov decision processes (DMDPs) fulfill the Markov property by augmenting the state space of agents with a finite time window of recently committed actions. In reliance on these state augmentations, delay-resolved reinforcement learning (RL) algorithms train policies to learn optimal interactions with environments featuring observation or action delays. Although such methods can be directly trained on the real robots, due to sample inefficiency, limited resources, or safety constraints, a common approach is to transfer models trained in simulation to the physical robot. However, robotic simulations rely on approximated models of the physical systems, which hinders the sim2real transfer. In this work, we consider various uncertainties in modeling the robot or environment dynamics as unknown intrinsic disturbances applied to the system input. We introduce the disturbance-augmented Markov decision process (DAMDP) in delayed settings as a novel representation to incorporate disturbance estimation in training on-policy RL algorithms. The proposed method is validated across several metrics on learning robotic reaching and pushing tasks and compared with disturbance-unaware baselines. The results show that the disturbance-augmented models can achieve higher stabilization and robustness in the control response, which in turn improves the prospects of successful sim2real transfer.Keywords
All Related Versions
Funding Information
- Research Project “A-IQ READY” within the Chips Joint Undertaking —the Public–Private Partnership for Research, Development, and Innovation
- HORIZON EUROPE Framework Programme
- National Authorities (101096658)
This publication has 28 references indexed in Scilit:
- Robust Tracking Control for Quadrotor UAV With External Disturbances and Uncertainties Using Neural Network Based MRACIEEE Access, 2024
- Delay-aware model-based reinforcement learning for continuous controlNeurocomputing, 2021
- Robotic box pushing under indeterminate anisotropic friction propertiesInternational Journal of Dynamics and Control, 2020
- Adapting Deep Visuomotor Representations with Weak Pairwise Constraints2020
- On Improving the Robustness of Reinforcement Learning-based Controllers using Disturbance Observer2019
- Leveraging Deep Reinforcement Learning for Reaching Robotic Tasks2017
- Automatic generation and detection of highly reliable fiducial markers under occlusionPattern Recognition, 2014
- Learning and planning in environments with delayed feedbackAutonomous Agents and Multi-Agent Systems, 2008
- H∞-Optimal Control and Related Minimax Design Problems2008
- Learning Without State-Estimation in Partially Observable Markovian Decision Processes1994