The Emergence of Large Language Models in Static Analysis: A First Look through Micro-Benchmarks
Open Access
- 14 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 35-39
Abstract
The application of Large Language Models (LLMs) in software engineering, particularly in static analysis tasks, represents a paradigm shift in the field. In this paper, we investigate the role that current LLMs can play in improving callgraph analysis and type inference for Python programs. Using the PyCG, HeaderGen, and TypeEvalPy micro-benchmarks, we evaluate 26 LLMs, including OpenAI's GPT series and open-source models such as LLaMA. Our study reveals that LLMs show promising results in type inference, demonstrating higher accuracy than traditional methods, yet they exhibit limitations in callgraph analysis. This contrast emphasizes the need for specialized fine-tuning of LLMs to better suit specific static analysis tasks. Our findings provide a foundation for further research towards integrating LLMs for static analysis tasks.Keywords
All Related Versions
Funding Information
- Ministry of Culture and Science of the State of North Rhine-Westphalia (NW21-059D)
This publication has 16 references indexed in Scilit:
- Assisting Static Analysis with Large Language Models: A ChatGPT Experiment2023
- The Hitchhiker's Guide to Program Analysis: A Journey with Large Language ModelsarXiv, 2023
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerarXiv, 2023
- A Survey on Model Compression for Large Language ModelsarXiv, 2023
- Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis2023
- The evolution of type annotations in python: an empirical study2022
- Static inference meets deep learning2022
- Type4Py2022
- PyCG: Practical Call Graph Generation in Python2021
- Typilus: neural type hints2020