Evaluating Large Language Models for Enhanced Fuzzing: An Analysis Framework for LLM-Driven Seed Generation
Open Access
- 23 October 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 156065-156081
Abstract
Fuzzing is a crucial technique for detecting software defects by dynamically generating and testing program inputs. This study introduces a framework designed to assess the application of Large Language Models (LLMs) to automate the generation of effective seed inputs for fuzzing, particularly in the Python programming environment where traditional approaches are less effective. Utilizing the Atheris fuzzing framework, we created over 38,000 seed inputs from LLMs targeted at 50 Python functions from widely-used libraries. Our findings underscore the critical role of LLM selection in seed effectiveness. In certain cases, seeds generated by LLMs rivaled or surpassed traditional fuzzing campaigns, with a corpus of fewer than 100 LLM-generated entries outperforming over 100,000 conventionally produced inputs. These seeds significantly improved code coverage and instruction count during fuzzing sessions, illustrating the efficacy of our framework in facilitating an automated, scalable approach to evaluating LLM effectiveness. The results, validated through linear regression analysis, demonstrate that selecting the appropriate LLM based on its training and capabilities is essential for optimizing fuzzing efficiency and facilitates the testing of future LLM versions.Keywords
This publication has 27 references indexed in Scilit:
- Automatic Program RepairIEEE Software, 2021
- Fuzzing: Challenges and ReflectionsIEEE Software, 2020
- File Guard: automatic format-based media file sanitizationInternational Journal of Information Security, 2019
- A systematic review of fuzzing techniquesComputers & Security, 2018
- A Coefficient of Determination for Generalized Linear ModelsThe American Statistician, 2017
- Learn&Fuzz: Machine learning for input fuzzing2017
- A Survey of Network Attacks Based on Protocol Vulnerabilities2014
- Have things changed now? An empirical study on input validation vulnerabilities in web applicationsComputers & Security, 2012
- Configuration Fuzzing for Software Vulnerability Detection2010
- Vulnerability Analysis for Mail Protocols2008