Gavin Black
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Gavin Black has a total of 8 co-authors. Here are the top co-authors by number of shared publications:
- Varghese Mathew Vaidyan
The Beacom College of Computer and Cyber Sciences, Dakota State University, Madison, SD, USA
3 shared publications - Gurcan Comert
North Carolina AT State University, Greensboro, NC, USA
2 shared publications - Bhaskar Prasad Rimal
Department of Computer Science, University of Idaho, Moscow, ID 83844, USA
1 shared publication - Eric Yocam
The Beacom College of Computer and Cyber Sciences, Dakota State University, Madison, SD, USA
1 shared publication - Kassandra Fronczyk
Leidos, Reston, VA, USA
1 shared publication Robert Allen
Leidos, Reston, VA, USA
1 shared publication
Latest Publications
- Open Access
- 23 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Data Descriptions
- Vol. 2, 163-172
Descriptor: Firewall Attack Detections and Extractions (FADE)
Validating defenses to meet emerging cybersecurity challenges requires continuous updates to the datasets used for testing. In this paper, we introduce the Firewall Attack Detection Extractions (FADE) dataset designed to address gaps in available collections by generating a diverse and balanced corpus of over 10 million categorized attacks derived from open-source rule sets and public penetration testing repositories. The FADE samples not only provide a wide variety of attacks across eight common categories but also combines these with realistic network traffic to create 50 million total entries, offering a balanced mix of labeled benign and malicious traffic released in both csv and Apache Parquet formats with rows containing HTTP request strings plus metadata columns that state the benign/malicious label, attack category, and byte-range of any injected payload. The methodology for dataset creation, along with the algorithms used for payload injection, is detailed to enhance reproducibility for future attack inclusions. The FADE dataset specifically addresses limitations of existing datasets by focusing on character-based request data suitable for language model testing, explicitly modeling firewall behavior with high-confidence labeled payloads, and providing payload offsets for granular detection testing. We also provide an exploratory data analysis that demonstrates the characteristics of the dataset, including similarities between attack token frequencies and embedding spaces, underscoring the challenges and considerations necessary for developing effective defensive security tools. A classification performance baseline is established using multiple methods, highlighting the difficulty in crafting suitable predictive models. The FADE dataset, along with the relevant tools for data analysis, is being publicly released to foster research and development in network security.
- 23 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Data Descriptions
- Vol. 2, 163-172
Open AccessDescriptor: Firewall Attack Detections and Extractions (FADE)
Validating defenses to meet emerging cybersecurity challenges requires continuous updates to the datasets used for testing. In this paper, we introduce the Firewall Attack Detection Extractions (FADE) dataset designed to address gaps in available collections by generating a diverse and balanced corpus of over 10 million categorized attacks derived from open-source rule sets and public penetration testing repositories. The FADE samples not only provide a wide variety of attacks across eight common categories but also combines these with realistic network traffic to create 50 million total entries, offering a balanced mix of labeled benign and malicious traffic released in both csv and Apache Parquet formats with rows containing HTTP request strings plus metadata columns that state the benign/malicious label, attack category, and byte-range of any injected payload. The methodology for dataset creation, along with the algorithms used for payload injection, is detailed to enhance reproducibility for future attack inclusions. The FADE dataset specifically addresses limitations of existing datasets by focusing on character-based request data suitable for language model testing, explicitly modeling firewall behavior with high-confidence labeled payloads, and providing payload offsets for granular detection testing. We also provide an exploratory data analysis that demonstrates the characteristics of the dataset, including similarities between attack token frequencies and embedding spaces, underscoring the challenges and considerations necessary for developing effective defensive security tools. A classification performance baseline is established using multiple methods, highlighting the difficulty in crafting suitable predictive models. The FADE dataset, along with the relevant tools for data analysis, is being publicly released to foster research and development in network security.
- Open Access
- 19 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 87928-87940
From LLMs to Randomness: Analyzing Program Input Efficacy With Resource and Language Metrics
Security-focused program testing typically focuses on crash detection and code coverage while overlooking additional system behaviors that can impact program confidentiality and availability. To address this gap, we propose a statistical framework that combines embedding-based anomaly detection, resource usage metrics, and resource-state distance measures to systematically profile software behaviors beyond traditional coverage-based methods. Leveraging over 5 million labeled samples from 50 Python programs, we evaluate how these independent scoring terms distinguish among different sources of input, including Large Language Model (LLM)-generated inputs, and demonstrate how standard statistical tests (e.g., Kolmogorov–Smirnov and Kendall’s τ) confirm their effectiveness. Our findings show that LLM-generated samples can trigger diverse behaviors but are often less effective at exploring resource usage dynamics (CPU, memory) compared with conventional fuzzing. However, combining LLM outputs with existing techniques broadens behavior coverage and reveals commonalities between commercial LLM outputs. We provide open-source tools for this evaluation framework, demonstrating the potential to refine software testing by integrating behavior metrics into security-testing workflows.
- 19 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 87928-87940
Open AccessFrom LLMs to Randomness: Analyzing Program Input Efficacy With Resource and Language Metrics
Security-focused program testing typically focuses on crash detection and code coverage while overlooking additional system behaviors that can impact program confidentiality and availability. To address this gap, we propose a statistical framework that combines embedding-based anomaly detection, resource usage metrics, and resource-state distance measures to systematically profile software behaviors beyond traditional coverage-based methods. Leveraging over 5 million labeled samples from 50 Python programs, we evaluate how these independent scoring terms distinguish among different sources of input, including Large Language Model (LLM)-generated inputs, and demonstrate how standard statistical tests (e.g., Kolmogorov–Smirnov and Kendall’s τ) confirm their effectiveness. Our findings show that LLM-generated samples can trigger diverse behaviors but are often less effective at exploring resource usage dynamics (CPU, memory) compared with conventional fuzzing. However, combining LLM outputs with existing techniques broadens behavior coverage and reveals commonalities between commercial LLM outputs. We provide open-source tools for this evaluation framework, demonstrating the potential to refine software testing by integrating behavior metrics into security-testing workflows.
- Open Access
- 23 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 156065-156081
Evaluating Large Language Models for Enhanced Fuzzing: An Analysis Framework for LLM-Driven Seed Generation
Fuzzing is a crucial technique for detecting software defects by dynamically generating and testing program inputs. This study introduces a framework designed to assess the application of Large Language Models (LLMs) to automate the generation of effective seed inputs for fuzzing, particularly in the Python programming environment where traditional approaches are less effective. Utilizing the Atheris fuzzing framework, we created over 38,000 seed inputs from LLMs targeted at 50 Python functions from widely-used libraries. Our findings underscore the critical role of LLM selection in seed effectiveness. In certain cases, seeds generated by LLMs rivaled or surpassed traditional fuzzing campaigns, with a corpus of fewer than 100 LLM-generated entries outperforming over 100,000 conventionally produced inputs. These seeds significantly improved code coverage and instruction count during fuzzing sessions, illustrating the efficacy of our framework in facilitating an automated, scalable approach to evaluating LLM effectiveness. The results, validated through linear regression analysis, demonstrate that selecting the appropriate LLM based on its training and capabilities is essential for optimizing fuzzing efficiency and facilitates the testing of future LLM versions.
- 23 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 156065-156081
Open AccessEvaluating Large Language Models for Enhanced Fuzzing: An Analysis Framework for LLM-Driven Seed Generation
Fuzzing is a crucial technique for detecting software defects by dynamically generating and testing program inputs. This study introduces a framework designed to assess the application of Large Language Models (LLMs) to automate the generation of effective seed inputs for fuzzing, particularly in the Python programming environment where traditional approaches are less effective. Utilizing the Atheris fuzzing framework, we created over 38,000 seed inputs from LLMs targeted at 50 Python functions from widely-used libraries. Our findings underscore the critical role of LLM selection in seed effectiveness. In certain cases, seeds generated by LLMs rivaled or surpassed traditional fuzzing campaigns, with a corpus of fewer than 100 LLM-generated entries outperforming over 100,000 conventionally produced inputs. These seeds significantly improved code coverage and instruction count during fuzzing sessions, illustrating the efficacy of our framework in facilitating an automated, scalable approach to evaluating LLM effectiveness. The results, validated through linear regression analysis, demonstrate that selecting the appropriate LLM based on its training and capabilities is essential for optimizing fuzzing efficiency and facilitates the testing of future LLM versions.
- 29 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 9 (1) , 419-430
Balancing Security and Correctness in Code Generation: An Empirical Study on Commercial Large Language Models
Large language models (LLMs) continue to be adopted for a multitude of previously manual tasks, with code generation as a prominent use. Multiple commercial models have seen wide adoption due to the accessible nature of the interface. Simple prompts can lead to working solutions that save developers time. However, the generated code has a significant challenge with maintaining security. There are no guarantees on code safety, and LLM responses can readily include known weaknesses. To address this concern, our research examines different prompt types for shaping responses from code generation tasks to produce safer outputs. The top set of common weaknesses is generated through unconditioned prompts to create vulnerable code across multiple commercial LLMs. These inputs are then paired with different contexts, roles, and identification prompts intended to improve security. Our findings show that the inclusion of appropriate guidance reduces vulnerabilities in generated code, with the choice of model having the most significant effect. Additionally, timings are presented to demonstrate the efficiency of singular requests that limit the number of model interactions.
- 29 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 9 (1) , 419-430
Balancing Security and Correctness in Code Generation: An Empirical Study on Commercial Large Language Models
Large language models (LLMs) continue to be adopted for a multitude of previously manual tasks, with code generation as a prominent use. Multiple commercial models have seen wide adoption due to the accessible nature of the interface. Simple prompts can lead to working solutions that save developers time. However, the generated code has a significant challenge with maintaining security. There are no guarantees on code safety, and LLM responses can readily include known weaknesses. To address this concern, our research examines different prompt types for shaping responses from code generation tasks to produce safer outputs. The top set of common weaknesses is generated through unconditioned prompts to create vulnerable code across multiple commercial LLMs. These inputs are then paired with different contexts, roles, and identification prompts intended to improve security. Our findings show that the inclusion of appropriate guidance reduces vulnerabilities in generated code, with the choice of model having the most significant effect. Additionally, timings are presented to demonstrate the efficiency of singular requests that limit the number of model interactions.