Segmenting Action-Value Functions Over Time-Scales in SARSA using TD($Δ$)
Author:
- Mahammad HumayooMHMahammad Humayoo
Preprint
Abstract
In numerous episodic reinforcement learning (RL) settings, SARSA-based methodologies are employed to enhance policies aimed at maximizing returns over long horizons. Conventional SARSA algorithms, however, have difficulties in balancing bias and variation due to the reliance on a singular, fixed discount factor. This study expands the temporal difference decomposition approach, TD($\triangle$), to the SARSA algorithm. SARSA, a widely utilised on-policy RL method, enhances action-value functions via temporal difference updates. TD($\triangle$) facilitates learning over several time-scales by breaking the action-value function into components associated with distinct discount factors. This decomposition improves learning efficiency and stability, particularly in problems necessitating long-horizon optimization. We illustrate that our methodology mitigates bias in SARSA's updates while facilitating accelerated convergence in contexts characterized by dense rewards. Experimental findings across many benchmark tasks indicate that the proposed SARSA($\triangle$) surpasses conventional TD learning methods in both tabular and deep RL contexts.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Gamma-Nets: Generalizing Value Estimation over TimescaleProceedings of the AAAI Conference on Artificial Intelligence, 2020
- OptionGAN: Learning Joint Reward-Policy Options Using Generative Adversarial Inverse Reinforcement LearningProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Near-Optimal Reinforcement Learning in Polynomial TimeMachine Learning, 2002
- Hierarchical Reinforcement Learning with the MAXQ Value Function DecompositionJournal of Artificial Intelligence Research, 2000