The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks
Preprint
Abstract
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Towards an understanding of large language models in software engineering tasksEmpirical Software Engineering, 2024
- Large Language Models for Software Engineering: A Systematic Literature ReviewACM Transactions on Software Engineering and Methodology, 2024
- A Survey on Model Compression for Large Language ModelsTransactions of the Association for Computational Linguistics, 2024
- Assisting Static Analysis with Large Language Models: A ChatGPT Experiment2023
- Large Language Models for Software Engineering: Survey and Open Problems2023
- Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis2023
- Static inference meets deep learning2022
- Type4Py2022
- PyCG: Practical Call Graph Generation in Python2021
- Typilus: neural type hints2020