Luolin Xiong

In cooperative Multi-Agent Reinforcement Learning (MARL), efficient exploration is crucial for optimizing the performance of joint policy. However, existing methods often update joint policies via independent agent exploration, without coordination among agents, which inherently constrains the expressive capacity and exploration of joint policies. To address this issue, we propose a conductor-based joint policy framework that directly enhances the expressive capacity of joint policies and coordinates exploration. In addition, we develop a Hierarchical Conductor-based Policy Optimization (HCPO) algorithm that instructs policy updates for the conductor and agents in a direction aligned with performance improvement. A rigorous theoretical guarantee further establishes the monotonicity of the joint policy optimization process. By deploying local conductors, HCPO retains centralized training benefits while eliminating inter-agent communication during execution. Finally, we evaluate HCPO on three challenging benchmarks: StarCraftII Multi-agent Challenge, Multi-agent MuJoCo, and Multi-agent Particle Environment. The results indicate that HCPO outperforms competitive MARL baselines regarding cooperative efficiency and stability.

With increasing penetration of Electric Vehicles (EVs) into the transportation system and smart electricity grid, there is a growing need for integrating them into Home Energy Management Systems (HEMS). This integration within HEMS introduces dynamic user behaviors and time-varying charging demand, thus posing challenges for the HEMS. To mitigate these challenges, this paper proposes a charging model for heterogeneous EVs that covers the range of Plug-in Hybrid EVs (PHEVs), Range-Extender EVs (REEVs) and Battery EVs (BEVs) with/without heat pumps. The proposed heterogeneous EV charging model considers weather conditions, estimated mileage and driver’s experience to describe the dynamic charging demand and the anxiety level influencing their behavior. To optimize the HEMS operation, minimizing the energy cost and ensuring comfort, this paper introduces an offline Deep Reinforcement Learning (DRL) algorithm which learns directly from pre-collected datasets, avoiding the cost and safety issues associated with continuous real-world interactions. The algorithm incorporates the Huber loss and a Q-quantile estimator to mitigate performance degradation from dataset anomalies such as data noise, sensor failure and human error, resulting in more robust HEMS optimization strategies. Experimental results demonstrate the method’s effectiveness in reducing total costs and analyze the performance of household devices with two different electricity rates.

DeepSeek, a Chinese Artificial Intelligence (AI) startup, has released their V3 and R1 series models, which attracted global attention due to their low cost, high performance, and open-source advantages. This paper begins by reviewing the evolution of large AI models focusing on paradigm shifts, the mainstream Large Language Model (LLM) paradigm, and the DeepSeek paradigm. Subsequently, the paper highlights novel algorithms introduced by DeepSeek, including Multi-head Latent Attention (MLA), Mixture-of-Experts (MoE), Multi-Token Prediction (MTP), and Group Relative Policy Optimization (GRPO). The paper then explores DeepSeek engineering breakthroughs in LLM scaling, training, inference, and system-level optimization architecture. Moreover, the impact of DeepSeek models on the competitive AI landscape is analyzed, comparing them to mainstream LLMs across various fields. Finally, the paper reflects on the insights gained from DeepSeek innovations and discusses future trends in the technical and engineering development of large AI models, particularly in data, training, and reasoning.

DeepSeek, a Chinese artificial intelligence (AI) startup, has released their V3 and R1 series models, which attracted global attention due to their low cost, high performance, and open-source advantages. This paper begins by reviewing the evolution of large AI models focusing on paradigm shifts, the mainstream large language model (LLM) paradigm, and the DeepSeek paradigm. Subsequently, the paper highlights novel algorithms introduced by DeepSeek, including multi-head latent attention (MLA), mixture-of-experts (MoE), multi-token prediction (MTP), and group relative policy optimization (GRPO). The paper then explores DeepSeek's engineering breakthroughs in LLM scaling, training, inference, and system-level optimization architecture. Moreover, the impact of DeepSeek models on the competitive AI landscape is analyzed, comparing them to mainstream LLMs across various fields. Finally, the paper reflects on the insights gained from DeepSeek's innovations and discusses future trends in the technical and engineering development of large AI models, particularly in data, training, and reasoning.

The increasing penetration of distributed energy resources has prompted distribution system operators (DSOs) at the retail electricity market level to coordinate with the independent system operator (ISO) at the wholesale market level, for greater benefits. However, interaction mechanisms between the ISO and DSOs, and impacts of prices and power injections, have not been adequately investigated in literature. This article proposes a distributed coordination framework for the ISO and DSOs across wholesale-retail (bi-level) electricity markets, considering their interactions more fairly. Moreover, to mitigate the challenges arising from the interdependence between the ISO and heterogeneous DSOs, a coupled training mechanism based on the response model is devised. This mechanism iteratively trains the ISO and DSOs by solely exchanging prices and power injections, ensuring the demand–supply balance at both retail and wholesale levels. In addition, a deep reinforcement learning algorithm is introduced for the three-stage iterative training process of heterogeneous agents. Results demonstrate the effectiveness of the proposed method and its advantages in terms of lowering energy prices, clearing of cheaper clean resources and thus, improving overall market efficiency.

Objective: The artificial pancreas (AP) has shown promising potential in achieving closed-loop glucose control for individuals with type 1 diabetes mellitus (T1DM). However, designing an effective control policy for the AP remains challenging due to the complex physiological processes, delayed insulin response, and inaccurate glucose measurements. While model predictive control (MPC) offers safety and stability through the dynamic model and safety constraints, it lacks individualization and is adversely affected by unannounced meals. Conversely, deep reinforcement learning (DRL) provides personalized and adaptive strategies but faces challenges with distribution shifts and substantial data requirements. Methods: We propose a hybrid control policy for the artificial pancreas (HyCPAP) to address the above challenges. HyCPAP combines an MPC policy with an ensemble DRL policy, leveraging the strengths of both policies while compensating for their respective limitations. To facilitate faster deployment of AP systems in real-world settings, we further incorporate meta-learning techniques into HyCPAP, leveraging previous experience and patient-shared knowledge to enable fast adaptation to new patients with limited available data. Results: We conduct extensive experiments using the FDA-accepted UVA/Padova T1DM simulator across three scenarios. Our approaches achieve the highest percentage of time spent in the desired euglycemic range and the lowest occurrences of hypoglycemia. Conclusion: The results clearly demonstrate the superiority of our methods for closed-loop glucose management in individuals with T1DM. Significance: The study presents novel control policies for AP systems, affirming the great potential of proposed methods for efficient closed-loop glucose control.

The precise localization of False Data Injection Attacks (FDIA) is vital to ensure the stable operation of smart grids. However, the intermittency and uncertainty of renewable energy can lead to confusion with unknown FDIA. As a result, previous works encountered difficulties in extracting distinguishable spatiotemporal features to construct accurate behavior models, thereby affecting the effectiveness of the localization task. To address this challenge, we establish a more practical dataset for FDIA localization that takes renewable energy into account. Subsequently, we propose a spatiotemporal sequence analysis framework for the task. Specifically, we propose a factorized module to mitigate the impact of temporal fluctuations, which processes data sequence with down sampling and feature aggregation. Additionally, we introduce a fine-tuning matrix to take regional correlations of renewable energy into consideration, where the weights of spatial information aggregation are adjusted. We evaluate the effectiveness of our approach through comprehensive case studies on IEEE 14-bus, IEEE 57-bus, and IEEE 118-bus standard test systems. The experimental results indicate that our method outperforms the compared methods by an average of 2.52% and 3% in terms of recall and F1-score, respectively.

Spatiotemporal forecasting in various domains, like traffic prediction and weather forecasting, is a challenging endeavor, primarily due to the difficulties in modeling propagation dynamics and capturing high-dimensional interactions among nodes. Despite the significant strides made by graph-based networks in spatiotemporal forecasting, there remain two pivotal factors closely related to forecasting performance that need further consideration: time delays in propagation dynamics and multi-scale high-dimensional interactions. In this work, we present a Series-Aligned Multi-Scale Graph Learning (SAMSGL) framework, aiming to enhance forecasting performance. In order to handle time delays in spatial interactions, we propose a series-aligned graph convolution layer to facilitate the aggregation of non-delayed graph signals, thereby mitigating the influence of time delays for the improvement in accuracy. To understand global and local spatiotemporal interactions, we develop a spatiotemporal architecture via multi-scale graph learning, which encompasses two essential components: multi-scale graph structure learning and graph-fully connected (Graph-FC) blocks. The multi-scale graph structure learning includes a global graph structure to learn both delayed and non-delayed node embeddings, as well as a local one to learn node variations influenced by neighboring factors. The Graph-FC blocks synergistically fuse spatial and temporal information to boost prediction accuracy. To evaluate the performance of SAMSGL, we conduct experiments on meteorological and traffic forecasting datasets, which demonstrate its effectiveness and superiority.