Segmenting Action-Value Functions over Time Scales in SARSA via TD(Δ)
Authors:
- Mahammad HumayooMHMahammad Humayoo
- Gengzhong ZhengGZGengzhong Zheng
- Xiaoqing DongXDXiaoqing Dong
- Wei HuangWHWei Huang
- Liming MiaoLMLiming Miao
- Shuwei QiuSQShuwei Qiu
- Zexun ZhouZZZexun Zhou
- Peitao WangPWPeitao Wang
- Zakir UllahZUZakir Ullah
- Naveed Ur Rehman JunejoNJNaveed Ur Rehman Junejo
- Xueqi ChengXCXueqi Cheng
Open Access
- 20 November 2025
- journal article
- research article
- Published by MDPI AG in Algorithms
- Vol. 18 (11) , 729
Abstract
In numerous episodic reinforcement learning (RL) environments, SARSA-based methodologies are employed to enhance policies aimed at maximizing returns over long horizons. Traditional SARSA algorithms face challenges in achieving an optimal balance between bias and variation, primarily due to their dependence on a single, constant discount factor (). This study enhances the temporal difference decomposition method, TD(), by applying it to the SARSA algorithm, wherein the action-value function is segmented into several components based on the differences between action-value functions linked to specific discount factors. Each component, referred to as a delta estimator (D), is linked to a specific discount factor and learned independently. This modified technique is referred to as SARSA(). SARSA is a widely used on-policy RL method that enhances action-value functions via temporal difference updates. This decomposition, namely SARSA(), facilitates learning across a range of time scales. This analysis makes learning more effective and guarantees consistency, especially in situations where long-horizon improvement is needed. The results of this research show that the proposed technique works to lower bias in SARSA’s updates and speed up convergence in both deterministic and stochastic settings, even in dense-reward Atari environments. Experimental results from a variety of benchmark settings show that the proposed SARSA() outperforms existing TD learning techniques in both tabular and deep RL environments.
Keywords
All Related Versions
Funding Information
- innovation teams of ordinary Universities in Guangdong Province (2021KCXTD038, 2023KCXTD022)
- Key Laboratory of Ordinary Universities in Guangdong Province (2022KSYS003)
- Key Discipline Research Ability Improvement Project of Guangdong Province (2021ZDJS043, 2022ZDJS068)
- Natural Science Foundation of Guangdong Province (2022A1515010990)
- Research Fund of the Department of Education of Guangdong Province (2022KTSCX079, 2023ZDZX1013, 2022ZDZX3012, 2022ZDZX3011, 2023ZDZX2038)
- Chaozhou Engineering Technology Research Center (z25025)
- Hanshan Normal University (XY202105)
This publication has 14 references indexed in Scilit:
- Multi-Horizon Multi-Agent Planning Using Decentralised Monte Carlo Tree SearchIEEE Robotics and Automation Letters, 2024
- Adaptive and multiple time-scale eligibility traces for online deep reinforcement learningRobotics and Autonomous Systems, 2022
- A theoretical and empirical analysis of Expected Sarsa2009
- Near-Optimal Reinforcement Learning in Polynomial TimeMachine Learning, 2002
- Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning2002
- An Overview of MAXQ Hierarchical Reinforcement Learning2000
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Adaptive critic designsIEEE Transactions on Neural Networks, 1997
- Learning to Predict by the Methods of Temporal DifferencesMachine Learning, 1988
- Use of Ranks in One-Criterion Variance AnalysisJournal of the American Statistical Association, 1952