PHPablo Hernandez-Leal

Pablo Hernandez-Leal

*This is an algorithmically generated author record based on publicly available information.

Pablo Hernandez-Leal has a total of 42 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Unobtrusive Stress Assessment Using Smartphones

    by Alban Maxhuni, Pablo Hernandez-Leal, Eduardo F. Morales, L. Enrique Sucar, Venet Osmani, Oscar Mayora

    Stress assessment is a complex issue and numerous studies have examined factors that influence stress in working environments. Research studies have shown that monitoring individuals' behaviour parameters during daily life can also help assess stress levels. In this study, we examine assessment of work-related stress using features derived from sensors in smartphones. In particular, we use information from physical activity levels, location, social-interactions, social-activity and application usage during working days. Our study included 30 employees chosen from two different private companies, monitored over a period of 8 weeks in real work environments. The findings suggest that information from phone sensors shows important correlation with employees perceived stress level. Secondly, we used machine learning methods to classify perceived stress levels based on the analysis of information provided by smartphones. We used decision trees obtaining 67.57% accuracy and 71.73% after applying a semi-supervised method. Our results show that stress levels can be monitored in unobtrusive manner, through analysis of smartphone data.


  • Top Cited Papers

    A survey and critique of multiagent deep reinforcement learning

    by Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor


  • Open Access

    Agent Modeling as Auxiliary Task for Deep Reinforcement Learning

    by Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor

    In this paper we explore how actor-critic methods in deep reinforcement learning, in particular Asynchronous Advantage Actor-Critic (A3C), can be extended with agent modeling. Inspired by recent works on representation learning and multiagent deep reinforcement learning, we propose two architectures to perform agent modeling: the first one based on parameter sharing, and the second one based on agent policy features. Both architectures aim to learn other agents’ policies as auxiliary tasks, besides the standard actor (policy) and critic (values). We performed experiments in both cooperative and competitive domains. The former is a problem of coordinated multiagent object transportation and the latter is a two-player mini version of the Pommerman game. Our results show that the proposed architectures stabilize learning and outperform the standard A3C architecture when learning a best response in terms of expected rewards.


  • Open Access

    A Survey and Critique of Multiagent Deep Reinforcement Learning

    by Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor

    Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.


  • Open Access

    Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learning

    by Bilal Kartal, Pablo Hernandez-Leal, Matthew E. Taylor

    Deep reinforcement learning has achieved great successes in recent years, but there are still open challenges, such as convergence to locally optimal policies and sample inefficiency. In this paper, we contribute a novel self-supervised auxiliary task, i.e., Terminal Prediction (TP), estimating temporal closeness to terminal states for episodic tasks. The intuition is to help representation learning by letting the agent predict how close it is to a terminal state, while learning its control policy. Although TP could be integrated with multiple algorithms, this paper focuses on Asynchronous Advantage Actor-Critic (A3C) and demonstrating the advantages of A3C-TP. Our extensive evaluation includes: a set of Atari games, the BipedalWalker domain, and a mini version of the recently proposed multi-agent Pommerman game. Our results on Atari games and the BipedalWalker domain suggest that A3C-TP outperforms standard A3C in most of the tested domains and in others it has similar performance. In Pommerman, our proposed method provides significant improvement both in learning efficiency and converging to better policies against different opponents.

  • Open Access

    Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition

    by Chao Gao, Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor

    The Pommerman Team Environment is a recently proposed benchmark which involves a multi-agent domain with challenges such as partial observability, decentralized execution (without communication), and very sparse and delayed rewards. The inaugural Pommerman Team Competition held at NeurIPS 2018 hosted 25 participants who submitted a team of 2 agents. Our submission nn_team_skynet955_skynet955 won 2nd place of the "learning agents'' category. Our team is composed of 2 neural networks trained with state of the art deep reinforcement learning algorithms and makes use of concepts like reward shaping, curriculum learning, and an automatic reasoning module for action pruning. Here, we describe these elements and additionally we present a collection of open-sourced agents that can be used for training and testing in the Pommerman environment. Code available at: https://github.com/BorealisAI/pommerman-baseline

  • Open Access

    Safer Deep RL with Shallow MCTS: A Case Study in Pommerman

    by Bilal Kartal, Pablo Hernandez-Leal, Chao Gao, Matthew E. Taylor

    Safe reinforcement learning has many variants and it is still an open research problem. Here, we focus on how to use action guidance by means of a non-expert demonstrator to avoid catastrophic events in a domain with sparse, delayed, and deceptive rewards: the recently-proposed multi-agent benchmark of Pommerman. This domain is very challenging for reinforcement learning (RL) --- past work has shown that model-free RL algorithms fail to achieve significant learning. In this paper, we shed light into the reasons behind this failure by exemplifying and analyzing the high rate of catastrophic events (i.e., suicides) that happen under random exploration in this domain. While model-free random exploration is typically futile, we propose a new framework where even a non-expert simulated demonstrator, e.g., planning algorithms such as Monte Carlo tree search with small number of rollouts, can be integrated to asynchronous distributed deep reinforcement learning methods. Compared to vanilla deep RL algorithms, our proposed methods both learn faster and converge to better policies on a two-player mini version of the Pommerman game.

  • Open Access

    A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity

    by Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, Enrique Munoz de Cote

    The key challenge in multiagent learning is learning a best response to the behaviour of other agents, which may be non-stationary: if the other agents adapt their strategy as well, the learning target moves. Disparate streams of research have approached non-stationarity from several angles, which make a variety of implicit assumptions that make it hard to keep an overview of the state of the art and to validate the innovation and significance of new works. This survey presents a coherent overview of work that addresses opponent-induced non-stationarity with tools from game theory, reinforcement learning and multi-armed bandits. Further, we reflect on the principle approaches how algorithms model and cope with this non-stationarity, arriving at a new framework and five categories (in increasing order of sophistication): ignore, forget, respond to target models, learn models, and theory of mind. A wide range of state-of-the-art algorithms is classified into a taxonomy, using these categories and key characteristics of the environment (e.g., observability) and adaptation behaviour of the opponents (e.g., smooth, abrupt). To clarify even further we present illustrative variations of one domain, contrasting the strengths and limitations of each category. Finally, we discuss in which environments the different approaches yield most merit, and point to promising avenues of future research.


  • Open Access

    An Exchange Mechanism to Coordinate Flexibility in Residential Energy Cooperatives

    by Shantanu Chakraborty, Pablo Hernandez-Leal, Michael Kaisers

    Energy cooperatives (ECs) such as residential and industrial microgrids have the potential to mitigate increasing fluctuations in renewable electricity generation, but only if their joint response is coordinated. However, the coordination and control of independently operated flexible resources (e.g., storage, demand response) imposes critical challenges arising from the heterogeneity of the resources, conflict of interests, and impact on the grid. Correspondingly, overcoming these challenges with a general and fair yet efficient exchange mechanism that coordinates these distributed resources will accommodate renewable fluctuations on a local level, thereby supporting the energy transition. In this paper, we introduce such an exchange mechanism. It incorporates a payment structure that encourages prosumers to participate in the exchange by increasing their utility above baseline alternatives. The allocation from the proposed mechanism increases the system efficiency (utilitarian social welfare) and distributes profits more fairly (measured by Nash social welfare) than individual flexibility activation. A case study analyzing the mechanism performance and resulting payments in numerical experiments over real demand and generation profiles of the Pecan Street dataset elucidates the efficacy to promote cooperation between co-located flexibilities in residential cooperatives through local exchange.

  • Coordinating Distributed and Flexible Resources: A Case-study of Residential Cooperatives

    by Shantanu Chakraborty, Pablo Hernandez-Leal, Michael Kaisers

    The increasing number of residential energy cooperatives raises the importance of forming a local prosumer marketplace that is capable of managing energy and flexibility exchange efficiently. However, the coordination and control of independently operated flexible resources (e.g. storage, demand response) imposes critical challenges arising from the heterogeneity of the flexible resources, conflict of interests, and impact on the grid. Therefore, designing a simple yet efficient coordination mechanism that works on these distributed resources is of the utmost importance. We introduce a simulation model to study energy exchange with flexibility coordination while working towards an efficient allocation mechanism. A case study analysing different allocation mechanisms and consequent losses (compared to a base-case of No flexibility) in numerical experiments over real demand/generation profiles of the Pecan Street dataset elucidates the efficacy in energy and flexibility allocation while promoting cooperation between co-located flexibilities in residential cooperatives through local exchange.