FAME: A Framework for Accelerating Independent Multi-Agent Reinforcement Learning on Heterogeneous Platforms
Open Access
- 13 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 471-484
Abstract
Multi-Agent Reinforcement Learning (MARL) enables multiple autonomous agents to learn and act in a shared environment. Independent learning (IL) is a widely used MARL paradigm that underpins many real-world applications requiring efficient training at scale. However, accelerating IL at scale is non-trivial. Existing MARL frameworks rely on single-process execution and homogeneous hardware assumptions, limiting scalability and underutilizing modern heterogeneous platforms composed of CPUs, GPUs, and FPGAs. Addressing this gap requires new execution models that increase parallelism while preserving IL training semantics. In this work, we present FAME, a framework that distributes computation across heterogeneous hardware resources while providing flexible interfaces that allow MARL practitioners to prototype and test new IL approaches. FAME is composed of: (1) high-level APIs that simplify IL algorithm development, (2) a heterogeneous IL training protocol that supports concurrent agent training on multiple diverse devices, while maintaining algorithm-agnostic training semantics, (3) automatic hardware configuration generation that optimizes system throughput without needing users to manually fine-tune their system setup, and (4) dynamic load balancing among devices with different compute and memory characteristics. We demonstrate FAME’s capabilities using three representative IL algorithms on a heterogeneous node platform consisting of CPUs, GPUs, and FPGAs. Implementations generated using FAME achieve a geometric mean end-to-end training time speedup of 7.1 × over state-of-the-art implementations and up to 2.7 × speedup over additional highly parallel baselines developed in this work.Keywords
Funding Information
- National Science Foundation (OAC-2411446)
This publication has 16 references indexed in Scilit:
- Stellaris: Staleness-Aware Distributed Reinforcement Learning with Serverless Computing2024
- Accelerating Multi-Agent DDPG on CPU-FPGA Heterogeneous Platform2023
- Orbit: A Unified Simulation Framework for Interactive Robot Learning EnvironmentsIEEE Robotics and Automation Letters, 2023
- A Framework for Mapping DRL Algorithms With Prioritized Replay Buffer Onto Heterogeneous PlatformsIEEE Transactions on Parallel and Distributed Systems, 2023
- LearningGroup: A Real-Time Sparse Training on FPGA via Learnable Weight Grouping for Multi-Agent Reinforcement Learning2022
- An FPGA-based multi-agent Reinforcement Learning timing synchronizerComputers and Electrical Engineering, 2022
- Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and ApplicationsIEEE Transactions on Cybernetics, 2020
- The multiple flying sidekicks traveling salesman problem: Parcel delivery with multiple dronesTransportation Research Part C: Emerging Technologies, 2020
- Optuna2019
- Monte Carlo and quasi-Monte Carlo methodsActa Numerica, 1998