Zhenyu Zhang

Concept drift is a phenomenon where the distribution of data streams changes over time. When this happens, model predictions become less accurate. Hence, models built in the past need to be re-learned for the current data. Two design questions need to be addressed in designing a strategy to re-learn models: which type of concept drift has occurred, and how to utilize the drift type to improve re-learning performance. Existing drift detection methods are often good at determining when drift has occurred. However, few retrieve information about how the drift came to be present in the stream. Hence, determining the impact of the type of drift on adaptation is difficult. Filling this gap, we designed a framework based on a lazy strategy called Type-Driven Lazy Drift Adaptor (Type-LDA). Type-LDA first retrieves information about both how and when a drift has occurred, then it uses this information to re-learn the new model. To identify the type of drift, a drift type identifier is pre-trained on synthetic data of known drift types. Furthermore, a drift point locator locates the optimal point of drift via a sharing loss. Hence, Type-LDA can select the optimal point, according to the drift type, to re-learn the new model. Experiments validate Type-LDA on both synthetic data and real-world data, and the results show that accurately identifying drift type can improve adaptation accuracy.

Experience memory, which stores and replays experience from the past for off-policy reinforcement learning (RL) agents, has become an essential component of cutting-edge algorithms in recent years. In most previous work, experience memory emphasizes recently acquired experiences while neglects long-ago ones, which causes high variance and catastrophic forgetting in continual learning. To solve the problem, we introduce the Staged Experience Mechanism(SEM) - a novel management mechanism of experience memory. SEM adaptively regulates the proportion of experiences based on the current learning stages, enabling agents not only to learn from new experiences but also from very old ones given limited memory capacity. In the experiment, SEM is coupled with two state-of-the-art off-policy algorithms compared with classic experience replay mechanisms on the suite of MuJoCo tasks. The result shows that RL with SEM is more stable and less variable than its counterparts in most tested continuous control environments, which reduces the training variance by 5% ∼ 104% while ensuring optimal scoring performance.

Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algorithm based on proximal policy optimization, mixed distributed proximal policy optimization (MDPPO), and show that it can accelerate and stabilize the training process. In our algorithm, multiple different policies train simultaneously and each of them controls several identical agents that interact with environments. Actions are sampled by each policy separately as usual, but the trajectories for the training process are collected from all agents, instead of only one policy. We find that if we choose some auxiliary trajectories elaborately to train policies, the algorithm will be more stable and quicker to converge especially in the environments with sparse rewards.