LXLi Xia

Li Xia

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Li Xia has a total of 70 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systems

    by Junkai Hu, Li Xia

    Efficiency and reliability are both crucial for energy management, especially in multi-microgrid systems (MMSs) integrating intermittent and distributed renewable energy sources. This study investigates an economic and reliable energy management problem in MMSs under a distributed scheme, where each microgrid independently updates its energy management policy in a decentralized manner to optimize the long-term system performance collaboratively. We introduce the mean and variance of the exchange power between the MMS and the main grid as indicators for the economic performance and reliability of the system. Accordingly, we formulate the energy management problem as a mean-variance team stochastic game (MV-TSG), where conventional methods based on the maximization of expected cumulative rewards are unsuitable for variance metrics. To solve MV-TSGs, we propose a fully distributed independent policy gradient algorithm, with rigorous convergence analysis, for scenarios with known model parameters. For large-scale scenarios with unknown model parameters, we further develop a deep reinforcement learning algorithm based on independent policy gradients, enabling data-driven policy optimization. Numerical experiments in two scenarios validate the effectiveness of the proposed methods. Our approaches fully leverage the distributed computational capabilities of MMSs and achieve a well-balanced trade-off between economic performance and operational reliability.

  • Open Access

    Sharpe Ratio Optimization in Markov Decision Processes

    by Shuai Ma, Guangwu Liu, Li Xia

    Sharpe ratio (also known as reward-to-variability ratio) is a widely-used metric in finance, which measures the additional return at the cost of per unit of increased risk (standard deviation of return). However, the optimization of Sharpe ratio in Markov decision processes (MDPs) is challenging, because there exist two difficulties hindering the application of dynamic programming. One is that dynamic programming does not work for fractional objectives, and the other is that dynamic programming is invalid for risk metrics. In this paper, we study the Sharpe ratio optimization in infinite-horizon MDPs, considering both the long-run average and discounted settings. We address the first challenge with the Dinkelbachs transform, which converts the Sharpe ratio objective to a mean-squared-variance (M2V) objective. It is shown that the M2V optimization and the original Sharpe ratio optimization share the same optimal policy when the risk-sensitive parameter is equal to the optimal Sharpe ratio. For the second challenge, we develop an iterative algorithm to solve the M2V optimization which is similar to a mean-variance optimization in MDPs. We iteratively solve the M2V problem and obtain the associated Sharpe ratio that is used to update the risk-sensitive parameter in the next iteration of M2V problems. We show that such a sequence of Sharpe ratios derived is monotonically increasing and converges to the optimal Sharpe ratio. For both average and discounted MDP settings, we develop a policy iteration procedure and prove its convergence to the optimum. Numerical experiments are conducted for validation. To the best of our knowledge, our approach is the first that solves the Sharpe ratio optimization in MDPs with dynamic programming type algorithms. We believe that the proposed algorithm can shed light on solving MDPs with other fractional objectives.

  • Open Access

    Markov Decision Processes with Value-at-Risk Criterion

    by Li Xia, Jinyan Pan

    Value-at-risk (VaR), also known as quantile, is a crucial risk measure in finance and other fields. However, optimizing VaR metrics in Markov decision processes (MDPs) is challenging because VaR is non-additive and the traditional dynamic programming is inapplicable. This paper conducts a comprehensive study on VaR optimization in discrete-time finite MDPs. We consider VaR in two key scenarios: the VaR of steady-state rewards over an infinite horizon and the VaR of accumulated rewards over a finite horizon. By establishing the equivalence between the VaR maximization MDP and a series of probabilistic minimization MDPs, we transform the VaR maximization MDP into a constrained bilevel optimization problem. The inner-level is a policy optimization of minimizing the probability that MDP rewards fall below a target $λ$, while the outer-level is a single parameter optimization of $λ$, representing the target VaR. For the steady-state scenario, the probabilistic minimization MDP can be resolved using the expectation criterion of a standard MDP. In contrast, for the finite-horizon case, it can be addressed via an augmented-state MDP. We prove the optimality of deterministic stationary policies for steady-state VaR MDPs and deterministic history-dependent policies for finite-horizon VaR MDPs. We derive both a policy improvement rule and a necessary-sufficient condition for optimal policies. Furthermore, we develop policy iteration type algorithms to maximize the VaR in MDPs and prove their convergence. Our results are also extended to the counterpart of VaR minimization MDPs after appropriate modifications. Finally, we conduct numerical experiments to demonstrate the computational efficiency and practical applicability of our approach. Our study paves a novel way to explore the optimization of quantile-related metrics in MDPs through the duality between quantiles and probabilities.

  • Open Access

    Team Variance Optimization of n-Player Stochastic Games with Separately Controlled Chains

    by Li Xia

    In this paper, we study a subclass of n-player stochastic games, in which each player has their own internal state controlled only by their own action and their objective is a common goal called team variance which measures the total variation of the random rewards of all players. It is assumed that players cannot observe each others' state/action. Thus, players' internal chains are controlled separately by their own action and they are coupled through the objective of team variance. Since the variance metric is not additive or Markovian, the dynamic programming principle fails in this problem. We study this problem from the viewpoint of sensitivity-based optimization. A difference formula and a derivative formula for team variance with respect to policy perturbations are derived, which provide sensitivity information to guide decentralized optimization. The existence of a stationary pure Nash equilibrium policy is derived. We further develop a bilevel optimization algorithm that iteratively updates the team mean at the outer level and minimizes the team variance at the inner level, where the team mean serves as a signal to coordinate the optimization of n players in a decentralized manner. We prove that the algorithm can converge to a strictly local minimum or a first-order stationary point in the space of mixed policies. Finally, we demonstrate the effectiveness of our approach using a numerical experiment of energy management in smart grid, where the assumption of separately controlled chains holds naturally.

  • Economical and Reliable Energy Management for Networked Microgrids in a Multi-Agent Collaborative Manner

    by Junkai Hu, Li Xia, Jianqiang Hu, Haoran Wu

    Reliability is a fundamental requirement of power systems. However, uncertainties from renewable energy generators and demand loads bring challenges to the economical and reliable operation of power distribution networks. This paper focuses on an energy management problem for networked microgrid systems (NMSs), aiming at establishing energy management policies for individual microgrids within NMSs to enhance the long-term economy and stability of the entire system. The key challenge is that each microgrid has independent decision-making capability and how to collaborate all microgrids’ decisions in a distributed multi-agent manner such that the entire system goal is optimized. We introduce the mean and variance of the exchanged power between the NMS and the main grid as the indicators of NMS operation profits and power fluctuations, respectively. The energy management problem is formulated as a mean-variance team stochastic game (MV-TSG) model. Because the variance metric is neither additive nor Markovian in a dynamic setting, the dynamic programming principle does not hold for this problem. We analyze the MV-TSG from a view of sensitivity-based optimization and propose a multi-agent policy iteration method by introducing a sequential update scheme. Furthermore, to tackle MV-TSGs with larger scales or unknown environmental parameters, we extend the idea of trust region optimization and develop a novel multi-agent deep reinforcement learning algorithm. The performance of our approaches is verified through numerical experiments using a real dataset in power distribution networks. Note to Practitioners —Energy management is essential for economical and reliable operation of networked microgrid systems (NMSs). NMSs exchange power with the main grid, where positive exchanged power indicates that the NMS is selling power to the main grid and generating profits, while negative exchanged power indicates that the NMS is purchasing power and incurring costs. The long-term average exchanged power between the NMS and the main grid reflects the economical operation of NMSs. However, fluctuations in exchanged power can lead to power quality issues such as frequency deviation and voltage flicker, which affect the reliability of NMS operations. In this paper, we study the economical and reliable operation of NMSs with distributed energy management policies, formulating the problem as a mean-variance team stochastic game (MV-TSG). We propose a multi-agent policy iteration method to address MV-TSGs. We further develop a multi-agent deep reinforcement learning algorithm to tackle MV-TSGs with large scales and in scenarios where environmental models are unknown. Experiments utilizing real data demonstrate that our algorithms effectively promote collaboration among microgrids to control exchanged power. As the weight coefficient increases, power fluctuations are reduced, achieving power smoothing and peak shaving simultaneously in long-term scheduling. Consequently, our algorithms can facilitate the trade-off between economy and reliability in NMS operations. These results provide valuable insights into the energy management challenges of NMSs in power distribution networks.


  • Open Access

    Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning (Extended Abstract)

    by Xiaoteng Ma, Shuai Ma, Li Xia, Qianchuan Zhao

    Electronic proceedings of IJCAI 2023

  • Open Access

    Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning

    by Xiaoteng Ma, Shuai Ma, Li Xia, Qianchuan Zhao

    Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, which penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures the negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.