Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation
Authors:
Top Cited Papers
- 28 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models2024
- Chip-Chat: Challenges and Opportunities in Conversational Hardware Design2023
- On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot2023
- Benchmarking Large Language Models for Automated Verilog RTL Code Generation2023
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2023
- Deduplicating Training Data Makes Language Models Better2022
- BioMegatron: Larger Biomedical Domain Language Model2020
- Pyverilog: A Python-Based Hardware Design Processing Toolkit for Verilog HDL2015
- On the resemblance and containment of documents2002
- BLEU2001