Esmaeil Seraj
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Esmaeil Seraj has a total of 63 co-authors. Here are the top co-authors by number of shared publications:
- Matthew Gombolay
School of Interactive Computing, Georgia Institute of Technology, Atlanta, USA
12 shared publications - Rohan Paleja
MIT Lincoln Laboratory, Lexington, Massachusetts, USA
7 shared publications - Letian Chen
Georgia Institute of Technology, Atlanta, GA, USA
4 shared publications - Reza Sameni
Emory University School of Medicine
4 shared publications Daniel Martin
Georgia Institute of Technology, Atlanta, GA, USA
2 shared publications- Kin Man Lee
Georgia Institute of Technology, Atlanta, GA, USA
2 shared publications
Latest Publications
- 16 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10  (2) , 1018-1025
Learning Multi-Agent Coordination for Replenishment At Sea
Optimizing large-scale logistics is computationally challenging due to its scale and requirement to be robust to stochastic and time-varying weather disturbances. However, prior research in multi-agent reinforcement learning (MARL) does not address scenarios that capture complexity of logistics operations influenced by dynamic weather patterns. To address this gap, we suggest a new MARL environment, MARINE that has two types of agents equipped with limited resources and integrates real wave data to model the influences of weather on the replenishment at sea (RAS) operation. To this end, we propose SchedHGNN, a novel MARL algorithm that incorporates a heterogeneous graph neural network and an intrinsic reward scheme to enhance agent coordination and mitigate challenges induced by environment non-stationarity. Our results show that the combination of effective RAS scheduling and improved communication enables our model to outperform competitive baselines by up to 37.8%. This achievement marks a significant advancement in applying MARL to complex, real-world logistics scenarios.
- 16 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10  (2) , 1018-1025
Learning Multi-Agent Coordination for Replenishment At Sea
Optimizing large-scale logistics is computationally challenging due to its scale and requirement to be robust to stochastic and time-varying weather disturbances. However, prior research in multi-agent reinforcement learning (MARL) does not address scenarios that capture complexity of logistics operations influenced by dynamic weather patterns. To address this gap, we suggest a new MARL environment, MARINE that has two types of agents equipped with limited resources and integrates real wave data to model the influences of weather on the replenishment at sea (RAS) operation. To this end, we propose SchedHGNN, a novel MARL algorithm that incorporates a heterogeneous graph neural network and an intrinsic reward scheme to enhance agent coordination and mitigate challenges induced by environment non-stationarity. Our results show that the combination of effective RAS scheduling and improved communication enables our model to outperform competitive baselines by up to 37.8%. This achievement marks a significant advancement in applying MARL to complex, real-world logistics scenarios.
- 22 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 40  (15523098) , 3833-3849
Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination
High-performing human-human teams learn intelligent and efficient communication and coordination strategies to maximize their joint utility. These teams implicitly understand the different roles of heterogeneous team members and adapt their communication protocols accordingly. Multi-Agent Reinforcement Learning (MARL) has attempted to develop computational methods for synthesizing such joint coordination-communication strategies, but emulating heterogeneous communication patterns across agents with different state, action, and observation spaces has remained a challenge. Without properly modeling agent heterogeneity, as in prior MARL work that leverages homogeneous graph networks, communication becomes less helpful and can even deteriorate the team's performance. In the past, we proposed Heterogeneous Policy Networks (HetNet) to learn efficient and diverse communication models for coordinating cooperative heterogeneous teams. In this extended work, we extend Heterogeneous Policy Networks (HetNet) to support scaling heterogeneous robot teams. Building on heterogeneous graph-attention networks, we show that HetNet not only facilitates learning heterogeneous collaborative policies but also enables end-to-end training for learning highly efficient binarized messaging. Our empirical evaluation shows that HetNet sets a new state of the art in learning coordination and communication strategies for heterogeneous multi-agent teams by achieving an 5.84% to 707.65% performance improvement over the next-best baseline across multiple domains while simultaneously achieving a 200× reduction in the required communication bandwidth.
- 22 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Robotics
- Vol. 40  (15523098) , 3833-3849
Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination
High-performing human-human teams learn intelligent and efficient communication and coordination strategies to maximize their joint utility. These teams implicitly understand the different roles of heterogeneous team members and adapt their communication protocols accordingly. Multi-Agent Reinforcement Learning (MARL) has attempted to develop computational methods for synthesizing such joint coordination-communication strategies, but emulating heterogeneous communication patterns across agents with different state, action, and observation spaces has remained a challenge. Without properly modeling agent heterogeneity, as in prior MARL work that leverages homogeneous graph networks, communication becomes less helpful and can even deteriorate the team's performance. In the past, we proposed Heterogeneous Policy Networks (HetNet) to learn efficient and diverse communication models for coordinating cooperative heterogeneous teams. In this extended work, we extend Heterogeneous Policy Networks (HetNet) to support scaling heterogeneous robot teams. Building on heterogeneous graph-attention networks, we show that HetNet not only facilitates learning heterogeneous collaborative policies but also enables end-to-end training for learning highly efficient binarized messaging. Our empirical evaluation shows that HetNet sets a new state of the art in learning coordination and communication strategies for heterogeneous multi-agent teams by achieving an 5.84% to 707.65% performance improvement over the next-best baseline across multiple domains while simultaneously achieving a 200× reduction in the required communication bandwidth.
- Open Access
- 5 June 2024
- Published by Emerald Publishing in Foundations and Trends in Robotics
- Vol. 12  (2-3) , 75-349
Interactive and Explainable Robot Learning: A Comprehensive Review
This review embarks on a comprehensive exploration of approaches, evaluation methods, and ethical considerations in explainable and interactive systems for robotic applications, distinctly focusing on intelligent systems that are specifically designed for learning automated agents. Given the increasing integration of robots in daily life, it is crucial to focus on intelligent systems that can not only learn and adapt, but can also offer clarity and comprehension for their actions. The interactive component of these systems is thoroughly examined, evaluating the algorithms, the modalities used in interaction, and the significance of mixed-initiative and shared autonomy. We spotlight adaptive and adaptable methods, emphasizing the centrality of user-inspired research and personalized approaches in interactive robotics. A rigorous examination of safety and ethical considerations of these intelligent systems anchors the discussion, including aspects of transparency, privacy, accountability, biases, and psychological well-being. The review evaluates existing metrics and benchmarking standards for such systems and explores their practical applications across domains such as healthcare, domestic tasks, and industrial automation. Concluding with key insights and directions for future research, we provide design guidelines and points of consensus for each subject in order to equip readers with a nuanced understanding of current trends and tools in explainable and interactive robotic systems, paving the way for informed research and application in this dynamic field.
- 5 June 2024
- Published by Emerald Publishing in Foundations and Trends in Robotics
- Vol. 12  (2-3) , 75-349
Open AccessInteractive and Explainable Robot Learning: A Comprehensive Review
This review embarks on a comprehensive exploration of approaches, evaluation methods, and ethical considerations in explainable and interactive systems for robotic applications, distinctly focusing on intelligent systems that are specifically designed for learning automated agents. Given the increasing integration of robots in daily life, it is crucial to focus on intelligent systems that can not only learn and adapt, but can also offer clarity and comprehension for their actions. The interactive component of these systems is thoroughly examined, evaluating the algorithms, the modalities used in interaction, and the significance of mixed-initiative and shared autonomy. We spotlight adaptive and adaptable methods, emphasizing the centrality of user-inspired research and personalized approaches in interactive robotics. A rigorous examination of safety and ethical considerations of these intelligent systems anchors the discussion, including aspects of transparency, privacy, accountability, biases, and psychological well-being. The review evaluates existing metrics and benchmarking standards for such systems and explores their practical applications across domains such as healthcare, domestic tasks, and industrial automation. Concluding with key insights and directions for future research, we provide design guidelines and points of consensus for each subject in order to equip readers with a nuanced understanding of current trends and tools in explainable and interactive robotic systems, paving the way for informed research and application in this dynamic field.
- 8 August 2023
- Published by Springer Nature in Current Robotics Reports
- Vol. 4  (3) , 81-100
Human-Robot Teaming: Grand Challenges
- 8 August 2023
- Published by Springer Nature in Current Robotics Reports
- Vol. 4  (3) , 81-100
Human-Robot Teaming: Grand Challenges
- Open Access
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37  (13) , 16135-16136
Embodied, Intelligent Communication for Multi-Agent Cooperation
High-performing human teams leverage intelligent and efficient communication and coordination strategies to collaboratively maximize their joint utility. Inspired by teaming behaviors among humans, I seek to develop computational methods for synthesizing intelligent communication and coordination strategies for collaborative multi-robot systems. I leverage both classical model-based control and planning approaches as well as data-driven methods such as Multi-Agent Reinforcement Learning (MARL) to provide several contributions towards enabling emergent cooperative teaming behavior across both homogeneous and heterogeneous (including agents with different capabilities) robot teams.
- 26 June 2023
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37  (13) , 16135-16136
Open AccessEmbodied, Intelligent Communication for Multi-Agent Cooperation
High-performing human teams leverage intelligent and efficient communication and coordination strategies to collaboratively maximize their joint utility. Inspired by teaming behaviors among humans, I seek to develop computational methods for synthesizing intelligent communication and coordination strategies for collaborative multi-robot systems. I leverage both classical model-based control and planning approaches as well as data-driven methods such as Multi-Agent Reinforcement Learning (MARL) to provide several contributions towards enabling emergent cooperative teaming behavior across both homogeneous and heterogeneous (including agents with different capabilities) robot teams.
- 10 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8  (4) , 2245-2252
Athletic Mobile Manipulator System for Robotic Wheelchair Tennis
Athletics are a quintessential and universal expression of humanity. From French monks who in the $12{\text{th}}$ century invented jeu de paume , the precursor to modern lawn tennis, back to the K'iche' people who played the Maya Ballgame as a form of religious expression over three thousand years ago, humans have sought to train their minds and bodies to excel in sporting contests. Advances in robotics are opening up the possibility of robots in sports. Yet, key challenges remain, as most prior works in robotics for sports are limited to pristine sensing environments, do not require significant force generation, or are on miniaturized scales unsuited for joint human-robot play. In this letter, we propose the first open-source, autonomous robot for playing regulation wheelchair tennis. We demonstrate the performance of our full-stack system in executing ground strokes and evaluate each of the system's hardware and software components. The goal of this letter is to (1) inspire more research in human-scale robot athletics and (2) establish the first baseline for a reproducible wheelchair tennis robot for regulation singles play. Our paper contributes to the science of systems design and poses a set of key challenges for the robotics community to address in striving towards robots that can match human capabilities in sports.
- 10 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8  (4) , 2245-2252
Athletic Mobile Manipulator System for Robotic Wheelchair Tennis
Athletics are a quintessential and universal expression of humanity. From French monks who in the $12{\text{th}}$ century invented jeu de paume , the precursor to modern lawn tennis, back to the K'iche' people who played the Maya Ballgame as a form of religious expression over three thousand years ago, humans have sought to train their minds and bodies to excel in sporting contests. Advances in robotics are opening up the possibility of robots in sports. Yet, key challenges remain, as most prior works in robotics for sports are limited to pristine sensing environments, do not require significant force generation, or are on miniaturized scales unsuited for joint human-robot play. In this letter, we propose the first open-source, autonomous robot for playing regulation wheelchair tennis. We demonstrate the performance of our full-stack system in executing ground strokes and evaluate each of the system's hardware and software components. The goal of this letter is to (1) inspire more research in human-scale robot athletics and (2) establish the first baseline for a reproducible wheelchair tennis robot for regulation singles play. Our paper contributes to the science of systems design and poses a set of key challenges for the robotics community to address in striving towards robots that can match human capabilities in sports.
- Open Access
Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming
Information sharing is key in building team cognition and enables coordination and cooperation. High-performing human teams also benefit from acting strategically with hierarchical levels of iterated communication and rationalizability, meaning a human agent can reason about the actions of their teammates in their decision-making. Yet, the majority of prior work in Multi-Agent Reinforcement Learning (MARL) does not support iterated rationalizability and only encourage inter-agent communication, resulting in a suboptimal equilibrium cooperation strategy. In this work, we show that reformulating an agent's policy to be conditional on the policies of its neighboring teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG). Building on the idea of decision-making under bounded rationality and cognitive hierarchy theory, we show that our modified PG approach not only maximizes local agent rewards but also implicitly reasons about MI between agents without the need for any explicit ad-hoc regularization terms. Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks. Our experiments validate the utility of InfoPG by achieving higher sample efficiency and significantly larger cumulative reward in several complex cooperative multi-agent domains.
Open AccessIterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming
Information sharing is key in building team cognition and enables coordination and cooperation. High-performing human teams also benefit from acting strategically with hierarchical levels of iterated communication and rationalizability, meaning a human agent can reason about the actions of their teammates in their decision-making. Yet, the majority of prior work in Multi-Agent Reinforcement Learning (MARL) does not support iterated rationalizability and only encourage inter-agent communication, resulting in a suboptimal equilibrium cooperation strategy. In this work, we show that reformulating an agent's policy to be conditional on the policies of its neighboring teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG). Building on the idea of decision-making under bounded rationality and cognitive hierarchy theory, we show that our modified PG approach not only maximizes local agent rewards but also implicitly reasons about MI between agents without the need for any explicit ad-hoc regularization terms. Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks. Our experiments validate the utility of InfoPG by achieving higher sample efficiency and significantly larger cumulative reward in several complex cooperative multi-agent domains.
- 24 June 2022
- Published by Springer Nature in Autonomous Agents and Multi-Agent Systems
- Vol. 36  (2) , 1-39
Multi-UAV planning for cooperative wildfire coverage and tracking with quality-of-service guarantees
- 24 June 2022
- Published by Springer Nature in Autonomous Agents and Multi-Agent Systems
- Vol. 36  (2) , 1-39
Multi-UAV planning for cooperative wildfire coverage and tracking with quality-of-service guarantees
- Open Access
- 1 June 2022
- Published by Office of Scientific and Technical Information (OSTI)
Scaling Multi-Agent Reinforcement Learning via State Upsampling.
We consider the problem of scaling Multi-Agent Reinforcement Learning (MARL) algorithms toward larger environments and team sizes. While it is possible to learn a MARL-synthesized policy on these larger problems from scratch, training is difficult as the joint state-action space is much larger. Policy learning will require a large amount of experience (and associated training time) to reach a target performance. In this paper, we propose a transfer learning method that accelerates the training performance in such high-dimensional tasks with increased complexity. Our method upsamples an agent's state representation in a smaller, less challenging, source task in order to pre-train a target policy for a larger, more challenging, target task. By transferring the policy after pre-training and continuing MARL in the target domain, the information learned within the source task enables higher performance within the target task in significantly less time than training from scratch. As such, our method enables the scalability of coordination problems. Furthermore, as our method only changes the state representation of agents across tasks, it is agnostic to the policy's architecture and can be deployed across different MARL algorithms. We provide results showing that a policy trained under our method is able to achieve up to a 7.88× performance improvement under the same amount of training time, compared to a policy trained from scratch. Moreover, our method enables learning in difficult target task settings where training from scratch fails.
- 1 June 2022
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessScaling Multi-Agent Reinforcement Learning via State Upsampling.
We consider the problem of scaling Multi-Agent Reinforcement Learning (MARL) algorithms toward larger environments and team sizes. While it is possible to learn a MARL-synthesized policy on these larger problems from scratch, training is difficult as the joint state-action space is much larger. Policy learning will require a large amount of experience (and associated training time) to reach a target performance. In this paper, we propose a transfer learning method that accelerates the training performance in such high-dimensional tasks with increased complexity. Our method upsamples an agent's state representation in a smaller, less challenging, source task in order to pre-train a target policy for a larger, more challenging, target task. By transferring the policy after pre-training and continuing MARL in the target domain, the information learned within the source task enables higher performance within the target task in significantly less time than training from scratch. As such, our method enables the scalability of coordination problems. Furthermore, as our method only changes the state representation of agents across tasks, it is agnostic to the policy's architecture and can be deployed across different MARL algorithms. We provide results showing that a policy trained under our method is able to achieve up to a 7.88× performance improvement under the same amount of training time, compared to a policy trained from scratch. Moreover, our method enables learning in difficult target task settings where training from scratch fails.
- Open Access
Heterogeneous Graph Attention Networks for Learning Diverse Communication
Multi-agent teaming achieves better performance when there is communication among participating agents allowing them to coordinate their actions for maximizing shared utility. However, when collaborating a team of agents with different action and observation spaces, information sharing is not straightforward and requires customized communication protocols, depending on sender and receiver types. Without properly modeling such heterogeneity in agents, communication becomes less helpful and could even deteriorate the multi-agent cooperation performance. We propose heterogeneous graph attention networks, called HetNet, to learn efficient and diverse communication models for coordinating heterogeneous agents towards accomplishing tasks that are of collaborative nature. We propose a Multi-Agent Heterogeneous Actor-Critic (MAHAC) learning paradigm to obtain collaborative per-class policies and effective communication protocols for composite robot teams. Our proposed framework is evaluated against multiple baselines in a complex environment in which agents of different types must communicate and cooperate to satisfy the objectives. Experimental results show that HetNet outperforms the baselines in learning sophisticated multi-agent communication protocols by achieving $\sim$10\% improvements in performance metrics.
Open AccessHeterogeneous Graph Attention Networks for Learning Diverse Communication
Multi-agent teaming achieves better performance when there is communication among participating agents allowing them to coordinate their actions for maximizing shared utility. However, when collaborating a team of agents with different action and observation spaces, information sharing is not straightforward and requires customized communication protocols, depending on sender and receiver types. Without properly modeling such heterogeneity in agents, communication becomes less helpful and could even deteriorate the multi-agent cooperation performance. We propose heterogeneous graph attention networks, called HetNet, to learn efficient and diverse communication models for coordinating heterogeneous agents towards accomplishing tasks that are of collaborative nature. We propose a Multi-Agent Heterogeneous Actor-Critic (MAHAC) learning paradigm to obtain collaborative per-class policies and effective communication protocols for composite robot teams. Our proposed framework is evaluated against multiple baselines in a complex environment in which agents of different types must communicate and cooperate to satisfy the objectives. Experimental results show that HetNet outperforms the baselines in learning sophisticated multi-agent communication protocols by achieving $\sim$10\% improvements in performance metrics.