Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities
Open Access
- 26 August 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 33 (7) , 1-29
Abstract
Code models have made significant advancements in code intelligence by encoding knowledge about programming languages. While previous studies have explored the capabilities of these models in learning code syntax, there has been limited investigation on their ability to understand code semantics. Additionally, existing analyses assume that the number of edges between nodes at the abstract syntax tree (AST) is related to syntax distance, and also often require transforming the high-dimensional space of deep learning models to a low-dimensional one, which may introduce inaccuracies. To study how code models represent code syntax and semantics, we conduct a comprehensive analysis of seven code models, including four representative code pre-trained models (CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder) and three large language models (LLMs) (StarCoder, CodeLlama and CodeT5+). We design four probing tasks to assess the models’ capacities in learning both code syntax and semantics. These probing tasks reconstruct code syntax and semantics structures (AST, control dependence graph (CDG), data dependence graph (DDG), and control flow graph (CFG)) in the representation space. These structures are core concepts for code understanding. We also investigate the syntax token role in each token representation and the long dependency between the code tokens. Additionally, we analyze the distribution of attention weights related to code semantic structures. Through extensive analysis, our findings highlight the strengths and limitations of different code models in learning code syntax and semantics. The results demonstrate that these models excel in learning code syntax, successfully capturing the syntax relationships between tokens and the syntax roles of individual tokens. However, their performance in encoding code semantics varies. CodeT5 and CodeBERT demonstrate proficiency in capturing control and data dependencies, whereas UnixCoder shows weaker performance in this aspect. We do not observe LLMs generally performing much better than pre-trained models. The shallow layers of LLMs perform better than their deep layers. The investigation of attention weights reveals that different attention heads play distinct roles in encoding code semantics. Our research findings emphasize the need for further enhancements in code models to better learn code semantics. This study contributes to the understanding of code models’ abilities in syntax and semantics analysis. Our findings provide guidance for future improvements in code models, facilitating their effective application in various code-related tasks.Keywords
All Related Versions
Funding Information
- National Research Foundation, Singapore
- National Cybersecurity R&D Programme (NCRP25-P04-TAICeN)
- National Research Foundation, Singapore
- DSO National Laboratories (AISG2-GC-2023-008)
- NRF Investigatorship (NRF-NRFI06-2020-0001)
- National Research Foundation, Singapore and Cyber Security Agency of Singapore
This publication has 41 references indexed in Scilit:
- Test Optimization in DNN Testing: A SurveyACM Transactions on Software Engineering and Methodology, 2024
- LaF: Labeling-free Model Selection for Automated Deep Neural Network ReusingACM Transactions on Software Engineering and Methodology, 2023
- CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2023
- A Survey on Deep Learning for Software EngineeringACM Computing Surveys, 2022
- Ranking Models in Unlabeled New Environments2021
- Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models2021
- Test Selection for Deep Learning SystemsACM Transactions on Software Engineering and Methodology, 2021
- A Primer in BERTology: What We Know About How BERT WorksTransactions of the Association for Computational Linguistics, 2020
- Fractional Norms and Quasinorms Do Not Help to Overcome the Curse of DimensionalityEntropy, 2020
- Comparison of the predicted and observed secondary structure of T4 phage lysozymeBiochimica et Biophysica Acta (BBA) - Protein Structure, 1975