Rujie Zhao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Rujie Zhao has a total of 8 co-authors. Here are the top co-authors by number of shared publications:
- Chao Lu
Southern Illinois Univ, Sch Elect Comp & Biomed Engn, Carbondale, IL 62901 USA
2 shared publications Lvqing Bi
Research Center for Intelligent Information and Communication Technology, School of Physics and Telecommunication Engineering, Yulin Normal University, Yulin, Guangxi, China
2 shared publications- Bike Xie
Kneron Inc., San Diego, CA, USA
1 shared publication - Defu Zhang
School of Informatics, Xiamen University, Xiamen, China
1 shared publication - Linkai Luo
Department of Automation, Xiamen University, Xiamen, China
1 shared publication - Tianlin Huang
College of Information and Smart Electromechanical Engineering, Xiamen Huaxia University, Xiamen 361024, China
1 shared publication
Latest Publications
- 25 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (10) , 7350-7364
Automatic Sparse Connectivity Learning for Neural Networks
Since sparse neural networks usually contain many zero weights, these unnecessary network connections can potentially be eliminated without degrading network performance. Therefore, well-designed sparse neural networks have the potential to significantly reduce the number of floating-point operations (FLOPs) and computational resources. In this work, we propose a new automatic pruning method--sparse connectivity learning (SCL). Specifically, a weight is reparameterized as an elementwise multiplication of a trainable weight variable and a binary mask. Thus, network connectivity is fully described by the binary mask, which is modulated by a unit step function. We theoretically prove the fundamental principle of using a straight-through estimator (STE) for network pruning. This principle is that the proxy gradients of STE should be positive, ensuring that mask variables converge at their minima. After finding Leaky ReLU, Softplus, and identity STEs can satisfy this principle, we propose to adopt identity STE in SCL for discrete mask relaxation. We find that mask gradients of different features are very unbalanced; hence, we propose to normalize mask gradients of each feature to optimize mask variable training. In order to automatically train sparse masks, we include the total number of network connections as a regularization term in our objective function. As SCL does not require pruning criteria or hyperparameters defined by designers for network layers, the network is explored in a larger hypothesis space to achieve optimized sparse connectivity for the best performance. SCL overcomes the limitations of existing automatic pruning methods. Experimental results demonstrate that SCL can automatically learn and select important network connections for various baseline network structures. Deep learning models trained by SCL outperform the state-of-the-art human-designed and automatic pruning methods in sparsity, accuracy, and FLOPs reduction.
- 25 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (10) , 7350-7364
Automatic Sparse Connectivity Learning for Neural Networks
Since sparse neural networks usually contain many zero weights, these unnecessary network connections can potentially be eliminated without degrading network performance. Therefore, well-designed sparse neural networks have the potential to significantly reduce the number of floating-point operations (FLOPs) and computational resources. In this work, we propose a new automatic pruning method--sparse connectivity learning (SCL). Specifically, a weight is reparameterized as an elementwise multiplication of a trainable weight variable and a binary mask. Thus, network connectivity is fully described by the binary mask, which is modulated by a unit step function. We theoretically prove the fundamental principle of using a straight-through estimator (STE) for network pruning. This principle is that the proxy gradients of STE should be positive, ensuring that mask variables converge at their minima. After finding Leaky ReLU, Softplus, and identity STEs can satisfy this principle, we propose to adopt identity STE in SCL for discrete mask relaxation. We find that mask gradients of different features are very unbalanced; hence, we propose to normalize mask gradients of each feature to optimize mask variable training. In order to automatically train sparse masks, we include the total number of network connections as a regularization term in our objective function. As SCL does not require pruning criteria or hyperparameters defined by designers for network layers, the network is explored in a larger hypothesis space to achieve optimized sparse connectivity for the best performance. SCL overcomes the limitations of existing automatic pruning methods. Experimental results demonstrate that SCL can automatically learn and select important network connections for various baseline network structures. Deep learning models trained by SCL outperform the state-of-the-art human-designed and automatic pruning methods in sparsity, accuracy, and FLOPs reduction.
- 29 April 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 33 (10) , 6021-6029
Neural Embedding Singular Value Decomposition for Collaborative Filtering
Singular value decomposition (SVD) is one of the most effective algorithms in recommender systems (RSs). Due to the iterative nature of SVD algorithms, one big challenge is initialization that has a major impact on the convergence and performance of RSs. Unfortunately, existing SVD algorithms in the literature typically initialize the user and item features in a random manner; thus, data information is not fully utilized. This work addresses the challenge of developing an efficient initialization method for SVD algorithms. We propose a general neural embedding initialization framework, where a low-complexity probabilistic autoencoder neural network initializes the features of user and item. This framework supports explicit and implicit feedback data sets. The design details of our proposed framework are elaborated and discussed. Experimental results show that RSs based on our proposed initialization framework outperform the state-of-the-art methods in rating prediction. Moreover, regarding item ranking, our proposed framework shows an improvement of at least 2.20% ~5.74% than existing SVD algorithms and other matrix factorization methods in the literature.
- 29 April 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 33 (10) , 6021-6029
Neural Embedding Singular Value Decomposition for Collaborative Filtering
Singular value decomposition (SVD) is one of the most effective algorithms in recommender systems (RSs). Due to the iterative nature of SVD algorithms, one big challenge is initialization that has a major impact on the convergence and performance of RSs. Unfortunately, existing SVD algorithms in the literature typically initialize the user and item features in a random manner; thus, data information is not fully utilized. This work addresses the challenge of developing an efficient initialization method for SVD algorithms. We propose a general neural embedding initialization framework, where a low-complexity probabilistic autoencoder neural network initializes the features of user and item. This framework supports explicit and implicit feedback data sets. The design details of our proposed framework are elaborated and discussed. Experimental results show that RSs based on our proposed initialization framework outperform the state-of-the-art methods in rating prediction. Moreover, regarding item ranking, our proposed framework shows an improvement of at least 2.20% ~5.74% than existing SVD algorithms and other matrix factorization methods in the literature.