Chao Jiang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Chao Jiang has a total of 4 co-authors. Here are the top co-authors by number of shared publications:
- Tianrun Lu
Jinling College, Nanjing University, Heibei, China
1 shared publication - Tianyi Hang
School of Information Science and Technology, Xi'an Jiaotong University, Xi'an, China
1 shared publication - Xinyi Xie
Faculty of Science and Technology, Beijing Normal University-Hong Kong Baptist University United International College, Zhuhai, China
1 shared publication - Yi Ding
School of Information Science and Technology, Xiamen University Tan Kah Kee College, Zhangzhou, Fujian, China
1 shared publication
Latest Publications
- Open Access
- 5 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 18-23
Enhancing HyperAttention: A Novel Approach for Improved Algorithmic Efficiency
HyperAttention is a new attention mechanism proposed to address the inevitability of quadratic time complexity and the conditionality of linear time when dealing with extended contexts, which allows the algorithm to maintain the computational complexity in linear time despite having large entries or a large stability rank in the matrix. HyperAttention improves the speed of inference and training of the large language model, with only a slight performance degradation and a significant speedup compared to FlashAttention. In order to further improve the accuracy of HyperAttention while maintaining its speed, we optimised each module for the original algorithm to improve its efficiency. Our main contribution is to change the locally sensitive hash algorithm used in HyperAttention for detecting large values and instead use k-means for indexing, further reducing the model's perplexity. In addition, we add applications on chatglm3-6b and baichuan-7b as well as chatglm3-6b-32k which supports long contexts in practice to make up for the lack of informativeness of HyperAttention in terms of practical applications, and test results are given. The experiments reveal that although the improved algorithm has a longer and relatively stable time complexity, it significantly reduces perplexity. Notably, with 10 to 15 replacement layers, the increase in time complexity is less than the reduction in perplexity, achieving a favourable balance between performance and efficiency. Moreover, we also analyse the feasibility of other sampling methods to improve accuracy and discuss the strategy for selecting the patch layers to apply the fast algorithm.
- 5 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 18-23
Open AccessEnhancing HyperAttention: A Novel Approach for Improved Algorithmic Efficiency
HyperAttention is a new attention mechanism proposed to address the inevitability of quadratic time complexity and the conditionality of linear time when dealing with extended contexts, which allows the algorithm to maintain the computational complexity in linear time despite having large entries or a large stability rank in the matrix. HyperAttention improves the speed of inference and training of the large language model, with only a slight performance degradation and a significant speedup compared to FlashAttention. In order to further improve the accuracy of HyperAttention while maintaining its speed, we optimised each module for the original algorithm to improve its efficiency. Our main contribution is to change the locally sensitive hash algorithm used in HyperAttention for detecting large values and instead use k-means for indexing, further reducing the model's perplexity. In addition, we add applications on chatglm3-6b and baichuan-7b as well as chatglm3-6b-32k which supports long contexts in practice to make up for the lack of informativeness of HyperAttention in terms of practical applications, and test results are given. The experiments reveal that although the improved algorithm has a longer and relatively stable time complexity, it significantly reduces perplexity. Notably, with 10 to 15 replacement layers, the increase in time complexity is less than the reduction in perplexity, achieving a favourable balance between performance and efficiency. Moreover, we also analyse the feasibility of other sampling methods to improve accuracy and discuss the strategy for selecting the patch layers to apply the fast algorithm.