Weixun Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Weixun Wang has a total of 64 co-authors. Here are the top co-authors by number of shared publications:
- Jianye Hao
Tianjin University, Tianjin, China
13 shared publications - Xiaotian Hao
College of Intelligence and Computing, Tianjin University
7 shared publications - Yujing Hu
NetEase Fuxi AI Lab
5 shared publications - Changjie Fan
NetEase Fuxi AI Lab, Hangzhou, China
4 shared publications - Tianpei Yang
College of Intelligence and Computing, Tianjin University, Tianjin, China
4 shared publications - Yingfeng Chen
Fuxi AI Lab in Netease
4 shared publications
Latest Publications
- Open Access
- 24 March 2024
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (14) , 15934-15942
PORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning
Despite many breakthroughs in recent years, it is still hard for MultiAgent Reinforcement Learning (MARL) algorithms to directly solve complex tasks in MultiAgent Systems (MASs) from scratch. In this work, we study how to use Automatic Curriculum Learning (ACL) to reduce the number of environmental interactions required to learn a good policy. In order to solve a difficult task, ACL methods automatically select a sequence of tasks (i.e., curricula). The idea is to obtain maximum learning progress towards the final task by continuously learning on tasks that match the current capabilities of the learners. The key question is how to measure the learning progress of the learner for better curriculum selection. We propose a novel ACL framework, PrOgRessive mulTiagent Automatic curricuLum (PORTAL), for MASs. PORTAL selects curricula according to two critera: 1) How difficult is a task, relative to the learners’ current abilities? 2) How similar is a task, relative to the final task? By learning a shared feature space between tasks, PORTAL is able to characterize different tasks based on the distribution of features and select those that are similar to the final task. Also, the shared feature space can effectively facilitate the policy transfer between curricula. Experimental results show that PORTAL can train agents to master extremely hard cooperative tasks, which can not be achieved with previous state-of-the-art MARL algorithms.
- 24 March 2024
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (14) , 15934-15942
Open AccessPORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning
Despite many breakthroughs in recent years, it is still hard for MultiAgent Reinforcement Learning (MARL) algorithms to directly solve complex tasks in MultiAgent Systems (MASs) from scratch. In this work, we study how to use Automatic Curriculum Learning (ACL) to reduce the number of environmental interactions required to learn a good policy. In order to solve a difficult task, ACL methods automatically select a sequence of tasks (i.e., curricula). The idea is to obtain maximum learning progress towards the final task by continuously learning on tasks that match the current capabilities of the learners. The key question is how to measure the learning progress of the learner for better curriculum selection. We propose a novel ACL framework, PrOgRessive mulTiagent Automatic curricuLum (PORTAL), for MASs. PORTAL selects curricula according to two critera: 1) How difficult is a task, relative to the learners’ current abilities? 2) How similar is a task, relative to the final task? By learning a shared feature space between tasks, PORTAL is able to characterize different tasks based on the distribution of features and select those that are similar to the final task. Also, the shared feature space can effectively facilitate the policy transfer between curricula. Experimental results show that PORTAL can train agents to master extremely hard cooperative tasks, which can not be achieved with previous state-of-the-art MARL algorithms.
- 10 November 2023
- Published by Springer Nature in Autonomous Agents and Multi-Agent Systems
- Vol. 37 (2) , 1-37
ASN: action semantics network for multiagent reinforcement learning
- 10 November 2023
- Published by Springer Nature in Autonomous Agents and Multi-Agent Systems
- Vol. 37 (2) , 1-37
ASN: action semantics network for multiagent reinforcement learning
- 13 September 2023
- Published by American Chemical Society (ACS) in Journal of the American Society for Mass Spectrometry
- Vol. 34 (10) , 2176-2186
Investigating Performance of the SLIM-Based High Resolution Ion Mobility Platform for Separation of Isomeric Phosphatidylcholine Species
Lipids are structurally diverse molecules that play a pivotal role in a plethora of biological processes. However, deciphering the biological roles of the specific lipids is challenging due to the existence of numerous isomers. This high chemical complexity of the lipidome is one of the major challenges in lipidomics research, as the traditional liquid chromatography-mass spectrometry (LC-MS) based approaches are often not powerful enough to resolve these isomeric and isobaric nuances within complex samples. Thus, lipids are uniquely suited to the benefits provided by multidimensional liquid chromatography-ion mobility-mass spectrometry (LC-IM-MS) analysis. However, many forms of lipid isomerism, including double-bond positional isomers and regioisomers, are structurally similar such that their collision cross section (CCS) differences are unresolvable via conventional IM approaches. Here we evaluate the performance of a high resolution ion mobility (HRIM) system based on structures for lossless ion manipulation (SLIM) technology interfaced to a high resolution quadrupole time-of-flight (QTOF) analyzer to address the noted lipidomic isomerism challenge. SLIM implements the traveling wave ion mobility technique along an ∼13 m ion path, providing longer path lengths to enable improved separation of isomeric features. We demonstrate the power of HRIM-MS to dissect isomeric PC standards differing only in double bond (DB) and stereospecific number (SN) positions. The partial separation of protonated DB isomers is significantly enhanced when they are analyzed as metal adducts. For sodium adducts, we achieve close to baseline separation of three different PC 18:1/18:1 isomers with different cis-double bond locations. Similarly, PC 18:1/18:1 (cis-9) can be resolved from the corresponding PC 18:1/18:1 (trans-9) form. The separation capacity is further enhanced when using silver ion doping, enabling the baseline separation of regioisomers that cannot be resolved when measured as sodium adducts. The sensitivity and reproducibility of the approach were assessed, and the performance for more complex mixtures was benchmarked by identifying PC isomers in total brain and liver lipid extracts.
- 13 September 2023
- Published by American Chemical Society (ACS) in Journal of the American Society for Mass Spectrometry
- Vol. 34 (10) , 2176-2186
Investigating Performance of the SLIM-Based High Resolution Ion Mobility Platform for Separation of Isomeric Phosphatidylcholine Species
Lipids are structurally diverse molecules that play a pivotal role in a plethora of biological processes. However, deciphering the biological roles of the specific lipids is challenging due to the existence of numerous isomers. This high chemical complexity of the lipidome is one of the major challenges in lipidomics research, as the traditional liquid chromatography-mass spectrometry (LC-MS) based approaches are often not powerful enough to resolve these isomeric and isobaric nuances within complex samples. Thus, lipids are uniquely suited to the benefits provided by multidimensional liquid chromatography-ion mobility-mass spectrometry (LC-IM-MS) analysis. However, many forms of lipid isomerism, including double-bond positional isomers and regioisomers, are structurally similar such that their collision cross section (CCS) differences are unresolvable via conventional IM approaches. Here we evaluate the performance of a high resolution ion mobility (HRIM) system based on structures for lossless ion manipulation (SLIM) technology interfaced to a high resolution quadrupole time-of-flight (QTOF) analyzer to address the noted lipidomic isomerism challenge. SLIM implements the traveling wave ion mobility technique along an ∼13 m ion path, providing longer path lengths to enable improved separation of isomeric features. We demonstrate the power of HRIM-MS to dissect isomeric PC standards differing only in double bond (DB) and stereospecific number (SN) positions. The partial separation of protonated DB isomers is significantly enhanced when they are analyzed as metal adducts. For sodium adducts, we achieve close to baseline separation of three different PC 18:1/18:1 isomers with different cis-double bond locations. Similarly, PC 18:1/18:1 (cis-9) can be resolved from the corresponding PC 18:1/18:1 (trans-9) form. The separation capacity is further enhanced when using silver ion doping, enabling the baseline separation of regioisomers that cannot be resolved when measured as sodium adducts. The sensitivity and reproducibility of the approach were assessed, and the performance for more complex mixtures was benchmarked by identifying PC isomers in total brain and liver lipid extracts.
- 2 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 16 (2) , 352-364
Cooperative Multiagent Transfer Learning With Coalition Pattern Decomposition
Knowledge transfer in cooperative multi-agent reinforcement learning (MARL) has drawn increasing attention in recent years. Unlike generalizing policies in single-agent tasks, it is more important to consider coordination knowledge than individual knowledge in multi-agent transfer learning. However, most of the existing methods only focus on knowledge transfer of the individual agent policy, which leads to coordination bias and finally affects the final performance in cooperative MARL. In this paper, we propose a level-adaptive MARL framework called “LA-QTransformer”, to realize the knowledge transfer on the coordination level via efficiently decomposing the agent coordination into multi-level coalition patterns for different agents. Compatible with centralized training with decentralized execution (CTDE) regime, LA-QTransformer utilizes the Level- Adaptive Transformer to generate suitable coalition patterns and then realizes the credit assignment for each agent. Besides, to deal with unexpected changes in the number of agents in the coordination transfer phase, we design a policy network called “Population invariant agent with Transformer (PIT)” to adapt dynamic observation and action space. We evaluate the LAQTransformer and PIT in the StarCraft II micro-management benchmark by comparing them with several state-of-the-art MARL baselines. The experimental results demonstrate the superiority of LA-QTransformer and PIT and verify the feasibility of coordination knowledge transfer.
- 2 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 16 (2) , 352-364
Cooperative Multiagent Transfer Learning With Coalition Pattern Decomposition
Knowledge transfer in cooperative multi-agent reinforcement learning (MARL) has drawn increasing attention in recent years. Unlike generalizing policies in single-agent tasks, it is more important to consider coordination knowledge than individual knowledge in multi-agent transfer learning. However, most of the existing methods only focus on knowledge transfer of the individual agent policy, which leads to coordination bias and finally affects the final performance in cooperative MARL. In this paper, we propose a level-adaptive MARL framework called “LA-QTransformer”, to realize the knowledge transfer on the coordination level via efficiently decomposing the agent coordination into multi-level coalition patterns for different agents. Compatible with centralized training with decentralized execution (CTDE) regime, LA-QTransformer utilizes the Level- Adaptive Transformer to generate suitable coalition patterns and then realizes the credit assignment for each agent. Besides, to deal with unexpected changes in the number of agents in the coordination transfer phase, we design a policy network called “Population invariant agent with Transformer (PIT)” to adapt dynamic observation and action space. We evaluate the LAQTransformer and PIT in the StarCraft II micro-management benchmark by comparing them with several state-of-the-art MARL baselines. The experimental results demonstrate the superiority of LA-QTransformer and PIT and verify the feasibility of coordination knowledge transfer.
- Open Access
Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL). Existing approaches such as potential-based reward shaping normally make full use of a given shaping reward function. However, since the transformation of human knowledge into numeric reward values is often imperfect due to reasons such as human cognitive bias, completely utilizing the shaping reward function may fail to improve the performance of RL algorithms. In this paper, we consider the problem of adaptively utilizing a given shaping reward function. We formulate the utilization of shaping rewards as a bi-level optimization problem, where the lower level is to optimize policy using the shaping rewards and the upper level is to optimize a parameterized shaping weight function for true reward maximization. We formally derive the gradient of the expected true reward with respect to the shaping weight function parameters and accordingly propose three learning algorithms based on different assumptions. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards, and meanwhile ignore unbeneficial shaping rewards or even transform them into beneficial ones.
Open AccessLearning to Utilize Shaping Rewards: A New Approach of Reward Shaping
Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL). Existing approaches such as potential-based reward shaping normally make full use of a given shaping reward function. However, since the transformation of human knowledge into numeric reward values is often imperfect due to reasons such as human cognitive bias, completely utilizing the shaping reward function may fail to improve the performance of RL algorithms. In this paper, we consider the problem of adaptively utilizing a given shaping reward function. We formulate the utilization of shaping rewards as a bi-level optimization problem, where the lower level is to optimize policy using the shaping rewards and the upper level is to optimize a parameterized shaping weight function for true reward maximization. We formally derive the gradient of the expected true reward with respect to the shaping weight function parameters and accordingly propose three learning algorithms based on different assumptions. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards, and meanwhile ignore unbeneficial shaping rewards or even transform them into beneficial ones.
- Open Access
- 1 July 2020
- Published by International Joint Conferences on Artificial Intelligence
- p. 3437-3443
Learning to Accelerate Heuristic Searching for Large-Scale Maximum Weighted b-Matching Problems in Online Advertising
Bipartite b-matching is fundamental in algorithm design, and has been widely applied into diverse applications, such as economic markets, labor markets, etc. These practical problems usually exhibit two distinct features: large-scale and dynamic, which requires the matching algorithm to be repeatedly executed at regular intervals. However, existing exact and approximate algorithms usually fail in such settings due to either requiring intolerable running time or too much computation resource. To address this issue, based on a key observation that the matching instances vary not too much, we propose NeuSearcher which leverage the knowledge learned from previously instances to solve new problem instances. Specifically, we design a multichannel graph neural network to predict the threshold of the matched edges, by which the search region could be significantly reduced. We further propose a parallel heuristic search algorithm to iteratively improve the solution quality until convergence. Experiments on both open and industrial datasets demonstrate that NeuSearcher can speed up 2 to 3 times while achieving exactly the same matching solution compared with the state-of-the-art approximation approaches.
- 1 July 2020
- Published by International Joint Conferences on Artificial Intelligence
- p. 3437-3443
Open AccessLearning to Accelerate Heuristic Searching for Large-Scale Maximum Weighted b-Matching Problems in Online Advertising
Bipartite b-matching is fundamental in algorithm design, and has been widely applied into diverse applications, such as economic markets, labor markets, etc. These practical problems usually exhibit two distinct features: large-scale and dynamic, which requires the matching algorithm to be repeatedly executed at regular intervals. However, existing exact and approximate algorithms usually fail in such settings due to either requiring intolerable running time or too much computation resource. To address this issue, based on a key observation that the matching instances vary not too much, we propose NeuSearcher which leverage the knowledge learned from previously instances to solve new problem instances. Specifically, we design a multichannel graph neural network to predict the threshold of the matched edges, by which the search region could be significantly reduced. We further propose a parallel heuristic search algorithm to iteratively improve the solution quality until convergence. Experiments on both open and industrial datasets demonstrate that NeuSearcher can speed up 2 to 3 times while achieving exactly the same matching solution compared with the state-of-the-art approximation approaches.
- Open Access
Efficient Deep Reinforcement Learning via Adaptive Policy Transfer
Transfer Learning (TL) has shown great potential to accelerate Reinforcement Learning (RL) by leveraging prior knowledge from past learned policies of relevant tasks. Existing transfer approaches either explicitly computes the similarity between tasks or select appropriate source policies to provide guided explorations for the target task. However, how to directly optimize the target policy by alternatively utilizing knowledge from appropriate source policies without explicitly measuring the similarity is currently missing. In this paper, we propose a novel Policy Transfer Framework (PTF) to accelerate RL by taking advantage of this idea. Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it by modeling multi-policy transfer as the option learning problem. PTF can be easily combined with existing deep RL approaches. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods in terms of learning efficiency and final performance in both discrete and continuous action spaces.
Open AccessEfficient Deep Reinforcement Learning via Adaptive Policy Transfer
Transfer Learning (TL) has shown great potential to accelerate Reinforcement Learning (RL) by leveraging prior knowledge from past learned policies of relevant tasks. Existing transfer approaches either explicitly computes the similarity between tasks or select appropriate source policies to provide guided explorations for the target task. However, how to directly optimize the target policy by alternatively utilizing knowledge from appropriate source policies without explicitly measuring the similarity is currently missing. In this paper, we propose a novel Policy Transfer Framework (PTF) to accelerate RL by taking advantage of this idea. Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it by modeling multi-policy transfer as the option learning problem. PTF can be easily combined with existing deep RL approaches. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods in terms of learning efficiency and final performance in both discrete and continuous action spaces.
- Open Access
KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge
Reinforcement learning agents usually learn from scratch, which requires a large number of interactions with the environment. This is quite different from the learning process of human. When faced with a new task, human naturally have the common sense and use the prior knowledge to derive an initial policy and guide the learning process afterwards. Although the prior knowledge may be not fully applicable to the new task, the learning process is significantly sped up since the initial policy ensures a quick-start of learning and intermediate guidance allows to avoid unnecessary exploration. Taking this inspiration, we propose knowledge guided policy network (KoGuN), a novel framework that combines human prior suboptimal knowledge with reinforcement learning. Our framework consists of a fuzzy rule controller to represent human knowledge and a refine module to fine-tune suboptimal prior knowledge. The proposed framework is end-to-end and can be combined with existing policy-based reinforcement learning algorithm. We conduct experiments on both discrete and continuous control tasks. The empirical results show that our approach, which combines human suboptimal knowledge and RL, achieves significant improvement on learning efficiency of flat RL algorithms, even with very low-performance human prior knowledge.
Open AccessKoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge
Reinforcement learning agents usually learn from scratch, which requires a large number of interactions with the environment. This is quite different from the learning process of human. When faced with a new task, human naturally have the common sense and use the prior knowledge to derive an initial policy and guide the learning process afterwards. Although the prior knowledge may be not fully applicable to the new task, the learning process is significantly sped up since the initial policy ensures a quick-start of learning and intermediate guidance allows to avoid unnecessary exploration. Taking this inspiration, we propose knowledge guided policy network (KoGuN), a novel framework that combines human prior suboptimal knowledge with reinforcement learning. Our framework consists of a fuzzy rule controller to represent human knowledge and a refine module to fine-tune suboptimal prior knowledge. The proposed framework is end-to-end and can be combined with existing policy-based reinforcement learning algorithm. We conduct experiments on both discrete and continuous control tasks. The empirical results show that our approach, which combines human suboptimal knowledge and RL, achieves significant improvement on learning efficiency of flat RL algorithms, even with very low-performance human prior knowledge.
- Open Access
Learning to Accelerate Heuristic Searching for Large-Scale Maximum Weighted b-Matching Problems in Online Advertising
Bipartite b-matching is fundamental in algorithm design, and has been widely applied into economic markets, labor markets, etc. These practical problems usually exhibit two distinct features: large-scale and dynamic, which requires the matching algorithm to be repeatedly executed at regular intervals. However, existing exact and approximate algorithms usually fail in such settings due to either requiring intolerable running time or too much computation resource. To address this issue, we propose \texttt{NeuSearcher} which leverages the knowledge learned from previously instances to solve new problem instances. Specifically, we design a multichannel graph neural network to predict the threshold of the matched edges weights, by which the search region could be significantly reduced. We further propose a parallel heuristic search algorithm to iteratively improve the solution quality until convergence. Experiments on both open and industrial datasets demonstrate that \texttt{NeuSearcher} can speed up 2 to 3 times while achieving exactly the same matching solution compared with the state-of-the-art approximation approaches.
Open AccessLearning to Accelerate Heuristic Searching for Large-Scale Maximum Weighted b-Matching Problems in Online Advertising
Bipartite b-matching is fundamental in algorithm design, and has been widely applied into economic markets, labor markets, etc. These practical problems usually exhibit two distinct features: large-scale and dynamic, which requires the matching algorithm to be repeatedly executed at regular intervals. However, existing exact and approximate algorithms usually fail in such settings due to either requiring intolerable running time or too much computation resource. To address this issue, we propose \texttt{NeuSearcher} which leverages the knowledge learned from previously instances to solve new problem instances. Specifically, we design a multichannel graph neural network to predict the threshold of the matched edges weights, by which the search region could be significantly reduced. We further propose a parallel heuristic search algorithm to iteratively improve the solution quality until convergence. Experiments on both open and industrial datasets demonstrate that \texttt{NeuSearcher} can speed up 2 to 3 times while achieving exactly the same matching solution compared with the state-of-the-art approximation approaches.
- Open Access
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous works borrow various multiagent coordination mechanisms into deep learning architecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different influences on other agents. In this paper, we propose a novel network architecture, named Action Semantics Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algorithms to boost their performance. Experimental results on StarCraft II micromanagement and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures.
Open AccessAction Semantics Network: Considering the Effects of Actions in Multiagent Systems
In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous works borrow various multiagent coordination mechanisms into deep learning architecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different influences on other agents. In this paper, we propose a novel network architecture, named Action Semantics Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algorithms to boost their performance. Experimental results on StarCraft II micromanagement and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures.