Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning
Open Access
- 6 November 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9092-9096
Abstract
LLM agents increasingly rely on reinforcement learning to improve multi-turn reasoning, tool use, and environment interaction. However, existing training pipelines often treat an agent trajectory as a flat token sequence or a reconstructed message trace, making it difficult to preserve rollout fidelity, manage long contexts, and inspect step-level credit signals. We present Agent-R1, an open-source modular framework for agentic reinforcement learning (RL). Agent-R1 uses interaction steps as the native unit for trajectory representation, replay, and environment interaction, and decouples exact trajectory storage from flexible context construction. The system supports context-management policies, step-level rollout inspection, action-mask visualization, tool-feedback analysis, and unified comparison of related RL algorithms. Through demonstrations on multiple tasks, Agent-R1 enables users to configure, train, inspect, and compare agentic RL workflows within a single interactive interface. Code and project resources are available at https://github.com/AgentR1/Agent-R1.Keywords
Funding Information
- National Natural Science Foundation of China (62502486)
- National Natural Science Foundation of China (U25B2072)
- Fundamental Research Funds for the Central Universities of China (N\/A)
- USTC Kunpeng-Ascend Scientific and Education Innovation Excellence Center (N\/A)
This publication has 7 references indexed in Scilit:
- A Comprehensive Survey of the LLM-Based Agent: The Contextual Cognition PerspectivePreprints.org, 2026
- TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoning2026
- HybridFlow: A Flexible and Efficient RLHF Framework2025
- AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning2025
- Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs2024
- Training Language Models to Follow Instructions with Human Feedback2022
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018