Zhen Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Zhen Wang has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Dacheng Tao
Nanyang technological university, Singapore, Singapore
4 shared publications - Liu Liu
School of Artificial Intelligence and Hangzhou International Innovation Institute, Beihang University, Beijing, China
3 shared publications - Yajing Kong
The University of Sydney
3 shared publications Bo Yuan
Shenzhen International Graduate School, Tsinghua University
2 shared publicationsYantao Yu
Shenzhen International Graduate School, Tsinghua University
2 shared publications- Bo Yuan
School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, QLD, Australia
1 shared publication
Latest Publications
- Open Access
- 28 September 2023
- Published by IOS Press
- p. 2680-2687
Adaptive Self-Supervised Continual Learning
Continual Learning (CL) studies the problem of developing a robust model that can learn new tasks while retaining previously learned knowledge. However, the current CL methods exclusively focus on data with annotations, disregarding that unlabelled data is the mainstream in real-world applications. To close this research gap, this study concentrates on continual self-supervised learning, which is plagued by challenges of memory over-fitting and class imbalance. Besides, these challenges are exacerbated throughout incremental training. Aimed at addressing these challenges from both loss and data perspectives, we introduce a framework, Adaptive Self-supervised Continual Learning (ASCL). Specifically, we devise an Adaptive Sharpness-Aware Minimization (ASAM) module responsible for identifying flatter local minima in the loss landscape with a smaller memory over-fitting risk. Additionally, we design an Adaptive Memory Enhancement (AME) module responsible for rebalancing self-supervised loss with new and old tasks from a data perspective. Finally, the adaptive mechanisms in AME and ASAM modules dynamically adjust the loss landscape sharpness and memory enhancement strength with the feedback of intermediate training results. The results of our extensive experiments demonstrate the state-of-the-art performance of our methods in continual self-supervised learning scenarios across multiple datasets.
- 28 September 2023
- Published by IOS Press
- p. 2680-2687
Open AccessAdaptive Self-Supervised Continual Learning
Continual Learning (CL) studies the problem of developing a robust model that can learn new tasks while retaining previously learned knowledge. However, the current CL methods exclusively focus on data with annotations, disregarding that unlabelled data is the mainstream in real-world applications. To close this research gap, this study concentrates on continual self-supervised learning, which is plagued by challenges of memory over-fitting and class imbalance. Besides, these challenges are exacerbated throughout incremental training. Aimed at addressing these challenges from both loss and data perspectives, we introduce a framework, Adaptive Self-supervised Continual Learning (ASCL). Specifically, we devise an Adaptive Sharpness-Aware Minimization (ASAM) module responsible for identifying flatter local minima in the loss landscape with a smaller memory over-fitting risk. Additionally, we design an Adaptive Memory Enhancement (AME) module responsible for rebalancing self-supervised loss with new and old tasks from a data perspective. Finally, the adaptive mechanisms in AME and ASAM modules dynamically adjust the loss landscape sharpness and memory enhancement strength with the feedback of intermediate training results. The results of our extensive experiments demonstrate the state-of-the-art performance of our methods in continual self-supervised learning scenarios across multiple datasets.
- Open Access
- 18 April 2023
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (7) , 1825-1839
Trust-Region Adaptive Frequency for Online Continual Learning
In the paradigm of online continual learning, one neural network is exposed to a sequence of tasks, where the data arrive in an online fashion and previously seen data are not accessible. Such online fashion causes insufficient learning and severe forgetting on past tasks issues, preventing a good stability-plasticity trade-off, where ideally the network is expected to have high plasticity to adapt to new tasks well and have the stability to prevent forgetting on old tasks simultaneously. To solve these issues, we propose a trust-region adaptive frequency approach, which alternates between standard-process and intra-process updates. Specifically, the standard-process replays data stored in a coreset and interleaves the data with current data, and the intra-process updates the network parameters based on the coreset. Furthermore, to improve the unsatisfactory performance stemming from online fashion, the frequency of the intra-process is adjusted based on a trust region, which is measured by the confidence score of current data. During the intra-process, we distill the dark knowledge to retain useful learned knowledge. Moreover, to store more representative data in the coreset, a confidence-based coreset selection is presented in an online manner. The experimental results on standard benchmarks show that the proposed method significantly outperforms state-of-art continual learning algorithms.
- 18 April 2023
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (7) , 1825-1839
Open AccessTrust-Region Adaptive Frequency for Online Continual Learning
In the paradigm of online continual learning, one neural network is exposed to a sequence of tasks, where the data arrive in an online fashion and previously seen data are not accessible. Such online fashion causes insufficient learning and severe forgetting on past tasks issues, preventing a good stability-plasticity trade-off, where ideally the network is expected to have high plasticity to adapt to new tasks well and have the stability to prevent forgetting on old tasks simultaneously. To solve these issues, we propose a trust-region adaptive frequency approach, which alternates between standard-process and intra-process updates. Specifically, the standard-process replays data stored in a coreset and interleaves the data with current data, and the intra-process updates the network parameters based on the coreset. Furthermore, to improve the unsatisfactory performance stemming from online fashion, the frequency of the intra-process is adjusted based on a trust region, which is measured by the confidence score of current data. During the intra-process, we distill the dark knowledge to retain useful learned knowledge. Moreover, to store more representative data in the coreset, a confidence-based coreset selection is presented in an online manner. The experimental results on standard benchmarks show that the proposed method significantly outperforms state-of-art continual learning algorithms.
- 1 November 2022
- Published by Springer Nature
- p. 219-236
Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning
- 1 November 2022
- Published by Springer Nature
- p. 219-236
Balancing Stability and Plasticity Through Advanced Null Space in Continual Learning
- 20 October 2022
- Published by Springer Nature
- p. 631-650
Online Continual Learning with Contrastive Vision Transformer
- 20 October 2022
- Published by Springer Nature
- p. 631-650
Online Continual Learning with Contrastive Vision Transformer
- 4 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (12) , 9952-9965
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning
Streaming label learning aims to model newly emerged labels for multilabel classification systems, which requires plenty of new label data for training. However, in changing environments, only a small amount of new label data can practically be collected. In this work, we formulate and study few-shot streaming label learning (FSLL), which models emerging new labels with only a few annotated examples by utilizing the knowledge learned from past labels. We propose a meta-learning framework, semantic inference network (SIN), which can learn and infer the semantic correlation between new labels and past labels to adapt FSLL tasks from a few examples effectively. SIN leverages label semantic representation to regularize the output space and acquires labelwise meta-knowledge based on gradient-based meta-learning. Moreover, SIN incorporates a novel label decision module with a meta-threshold loss to find the optimal confidence thresholds for each new label. Theoretically, we illustrate that the proposed semantic inference mechanism could constrain the complexity of hypotheses space to reduce the risk of overfitting and achieve better generalizability. Experimentally, extensive empirical results and ablation studies demonstrate the performance of SIN is superior to the prior state-of-the-art methods on FSLL.
- 4 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (12) , 9952-9965
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning
Streaming label learning aims to model newly emerged labels for multilabel classification systems, which requires plenty of new label data for training. However, in changing environments, only a small amount of new label data can practically be collected. In this work, we formulate and study few-shot streaming label learning (FSLL), which models emerging new labels with only a few annotated examples by utilizing the knowledge learned from past labels. We propose a meta-learning framework, semantic inference network (SIN), which can learn and infer the semantic correlation between new labels and past labels to adapt FSLL tasks from a few examples effectively. SIN leverages label semantic representation to regularize the output space and acquires labelwise meta-knowledge based on gradient-based meta-learning. Moreover, SIN incorporates a novel label decision module with a meta-threshold loss to find the optimal confidence thresholds for each new label. Theoretically, we illustrate that the proposed semantic inference mechanism could constrain the complexity of hypotheses space to reduce the risk of overfitting and achieve better generalizability. Experimentally, extensive empirical results and ablation studies demonstrate the performance of SIN is superior to the prior state-of-the-art methods on FSLL.
- 5 May 2021
- Published by Taylor & Francis in Transportation Letters
- Vol. 14 (6) , 660-674
Differentiable road pricing for environment-oriented electric vehicle and gasoline vehicle users in the bi-objective transportation network
The environmental traffic assignment problem in the bimodal network with electric vehicles (EVs) and gasoline vehicles (GVs) has become a hot topic recently. However, few previous works consider the psychological difference between EV users and GV users in terms of environmental awareness , in order to fill in this research gap, we formulate an environment-oriented mixed-behavior user equilibrium (EMUE) model to distinguish the route choice criterion between EV users and GV users by considering the Pro-environmental behavior (PEB) of EV users. . We formulate a bi-objective optimization model and propose a differentiable road pricing scheme for EV and GV users to manage the congestion and emissions simultaneously. We investigate whether the differentiable road pricing can decentralize the Pareto-efficient flow as a unique mixed-behavior equilibrium. Moreover, we conduct numerical experiments based on the Sioux Falls network and the existing urban network in Qianhai Zone to evaluate the effectiveness of the proposed pricing scheme, models and algorithms.
- 5 May 2021
- Published by Taylor & Francis in Transportation Letters
- Vol. 14 (6) , 660-674
Differentiable road pricing for environment-oriented electric vehicle and gasoline vehicle users in the bi-objective transportation network
The environmental traffic assignment problem in the bimodal network with electric vehicles (EVs) and gasoline vehicles (GVs) has become a hot topic recently. However, few previous works consider the psychological difference between EV users and GV users in terms of environmental awareness , in order to fill in this research gap, we formulate an environment-oriented mixed-behavior user equilibrium (EMUE) model to distinguish the route choice criterion between EV users and GV users by considering the Pro-environmental behavior (PEB) of EV users. . We formulate a bi-objective optimization model and propose a differentiable road pricing scheme for EV and GV users to manage the congestion and emissions simultaneously. We investigate whether the differentiable road pricing can decentralize the Pareto-efficient flow as a unique mixed-behavior equilibrium. Moreover, we conduct numerical experiments based on the Sioux Falls network and the existing urban network in Qianhai Zone to evaluate the effectiveness of the proposed pricing scheme, models and algorithms.
- 1 July 2020
- Published by International Joint Conferences on Artificial Intelligence
- p. 3139-3145
A Dual Input-aware Factorization Machine for CTR Prediction
Factorization Machines (FMs) refer to a class of general predictors working with real valued feature vectors, which are well-known for their ability to estimate model parameters under significant sparsity and have found successful applications in many areas such as the click-through rate (CTR) prediction. However, standard FMs only produce a single fixed representation for each feature across different input instances, which may limit the CTR model’s expressive and predictive power. Inspired by the success of Input-aware Factorization Machines (IFMs), which aim to learn more flexible and informative representations of a given feature according to different input instances, we propose a novel model named Dual Input-aware Factorization Machines (DIFMs) that can adaptively reweight the original feature representations at the bit-wise and vector-wise levels simultaneously. Furthermore, DIFMs strategically integrate various components including Multi-Head Self-Attention, Residual Networks and DNNs into a unified end-to-end model. Comprehensive experiments on two real-world CTR prediction datasets show that the DIFM model can outperform several state-of-the-art models consistently.
- 1 July 2020
- Published by International Joint Conferences on Artificial Intelligence
- p. 3139-3145
A Dual Input-aware Factorization Machine for CTR Prediction
Factorization Machines (FMs) refer to a class of general predictors working with real valued feature vectors, which are well-known for their ability to estimate model parameters under significant sparsity and have found successful applications in many areas such as the click-through rate (CTR) prediction. However, standard FMs only produce a single fixed representation for each feature across different input instances, which may limit the CTR model’s expressive and predictive power. Inspired by the success of Input-aware Factorization Machines (IFMs), which aim to learn more flexible and informative representations of a given feature according to different input instances, we propose a novel model named Dual Input-aware Factorization Machines (DIFMs) that can adaptively reweight the original feature representations at the bit-wise and vector-wise levels simultaneously. Furthermore, DIFMs strategically integrate various components including Multi-Head Self-Attention, Residual Networks and DNNs into a unified end-to-end model. Comprehensive experiments on two real-world CTR prediction datasets show that the DIFM model can outperform several state-of-the-art models consistently.
- 1 August 2019
- Published by International Joint Conferences on Artificial Intelligence
- p. 1466-1472
An Input-aware Factorization Machine for Sparse Prediction
Factorization machines (FMs) are a class of general predictors working effectively with sparse data, which represents features using factorized parameters and weights. However, the accuracy of FMs can be adversely affected by the fixed representation trained for each feature, as the same feature is usually not equally predictive and useful in different instances. In fact, the inaccurate representation of features may even introduce noise and degrade the overall performance. In this work, we improve FMs by explicitly considering the impact of individual input upon the representation of features. We propose a novel model named \textit{Input-aware Factorization Machine} (IFM), which learns a unique input-aware factor for the same feature in different instances via a neural network. Comprehensive experiments on three real-world recommendation datasets are used to demonstrate the effectiveness and mechanism of IFM. Empirical results indicate that IFM is significantly better than the standard FM model and consistently outperforms four state-of-the-art deep learning based methods.
- 1 August 2019
- Published by International Joint Conferences on Artificial Intelligence
- p. 1466-1472
An Input-aware Factorization Machine for Sparse Prediction
Factorization machines (FMs) are a class of general predictors working effectively with sparse data, which represents features using factorized parameters and weights. However, the accuracy of FMs can be adversely affected by the fixed representation trained for each feature, as the same feature is usually not equally predictive and useful in different instances. In fact, the inaccurate representation of features may even introduce noise and degrade the overall performance. In this work, we improve FMs by explicitly considering the impact of individual input upon the representation of features. We propose a novel model named \textit{Input-aware Factorization Machine} (IFM), which learns a unique input-aware factor for the same feature in different instances via a neural network. Comprehensive experiments on three real-world recommendation datasets are used to demonstrate the effectiveness and mechanism of IFM. Empirical results indicate that IFM is significantly better than the standard FM model and consistently outperforms four state-of-the-art deep learning based methods.
- 1 April 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 280-291
DBSVEC: Density-Based Clustering Using Support Vector Expansion
DBSCAN is a popular clustering algorithm that can discover clusters of arbitrary shapes with broad applications. However, DBSCAN is computationally expensive, as it performs range queries for all the points to determine their neighbors and grow the clusters. To address this problem, we propose a novel approximate density-based clustering algorithm named DBSVEC. DBSVEC introduces support vectors into density-based clustering, which allows performing range queries only on a small subset of points called the core support vectors. This technique significantly improves the efficiency while retaining high-quality cluster results. We evaluate the performance of DBSVEC via extensive experiments on real and synthetic datasets. The results show that DBSVEC is up to three orders of magnitude faster than DBSCAN. Compared with the state-of-the-art approximate density-based clustering methods, DBSVEC is up to two orders of magnitude faster, and the clustering results of DBSVEC are more similar to those of DBSCAN.
- 1 April 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 280-291
DBSVEC: Density-Based Clustering Using Support Vector Expansion
DBSCAN is a popular clustering algorithm that can discover clusters of arbitrary shapes with broad applications. However, DBSCAN is computationally expensive, as it performs range queries for all the points to determine their neighbors and grow the clusters. To address this problem, we propose a novel approximate density-based clustering algorithm named DBSVEC. DBSVEC introduces support vectors into density-based clustering, which allows performing range queries only on a small subset of points called the core support vectors. This technique significantly improves the efficiency while retaining high-quality cluster results. We evaluate the performance of DBSVEC via extensive experiments on real and synthetic datasets. The results show that DBSVEC is up to three orders of magnitude faster than DBSCAN. Compared with the state-of-the-art approximate density-based clustering methods, DBSVEC is up to two orders of magnitude faster, and the clustering results of DBSVEC are more similar to those of DBSCAN.