LLM vs. SAST: A Technical Analysis on Detecting Coding Bugs of GPT4-Advanced Data Analysis
Preprint
Abstract
With the rapid advancements in Natural Language Processing (NLP), large language models (LLMs) like GPT-4 have gained significant traction in diverse applications, including security vulnerability scanning. This paper investigates the efficacy of GPT-4 in identifying software vulnerabilities compared to traditional Static Application Security Testing (SAST) tools. Drawing from an array of security mistakes, our analysis underscores the potent capabilities of GPT-4 in LLM-enhanced vulnerability scanning. We unveiled that GPT-4 (Advanced Data Analysis) outperforms SAST by an accuracy of 94% in detecting 32 types of exploitable vulnerabilities. This study also addresses the potential security concerns surrounding LLMs, emphasising the imperative of security by design/default and other security best practices for AI.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Towards provably efficient quantum algorithms for large-scale machine-learning modelsNature Communications, 2024
- The FormAI Dataset: Generative AI in Software Security through the Lens of Formal Verification2023
- DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection2023
- I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and DefencesACM Computing Surveys, 2023
- An Analysis of the Automatic Bug Fixing Performance of ChatGPT2023
- A systematic evaluation of large language models of code2022
- Membership Inference Attacks Against Machine Learning Models2017
- FirePatch: Secure and Time-Critical Dissemination of Software Patches2007
- Reflections on trusting trustCommunications of the ACM, 1984
- Note on the Sampling Error of the Difference Between Correlated Proportions or PercentagesPsychometrika, 1947