WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Top Cited Papers
Open Access
- 8 October 2024
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Programming Languages
- Vol. 8 (OOPSLA2) , 709-735
Abstract
Compiler correctness is crucial, as miscompilation can falsify program behaviors, leading to serious consequences over the software supply chain. In the literature, fuzzing has been extensively studied to uncover compiler defects. However, compiler fuzzing remains challenging: Existing arts focus on black- and grey-box fuzzing, which generates test programs without sufficient understanding of internal compiler behaviors. As such, they often fail to construct test programs to exercise intricate optimizations. Meanwhile, traditional white-box techniques, such as symbolic execution, are computationally inapplicable to the giant codebase of compiler systems. Recent advances demonstrate that Large Language Models (LLMs) excel in code generation/understanding tasks and even have achieved state-of-the-art performance in black-box fuzzing. Nonetheless, guiding LLMs with compiler source-code information remains a missing piece of research in compiler testing. To this end, we propose W hite F ox , the first white-box compiler fuzzer using LLMs with source-code information to test compiler optimization, with a spotlight on detecting deep logic bugs in the emerging deep learning (DL) compilers. W hite F ox adopts a multi-agent framework: (i) an LLM-based analysis agent examines the low-level optimization source code and produces requirements on the high-level test programs that can trigger the optimization; (ii) an LLM-based generation agent produces test programs based on the summarized requirements. Additionally, optimization-triggering tests are also used as feedback to further enhance the test generation prompt on the fly. Our evaluation on the three most popular DL compilers ( i.e ., PyTorch Inductor, TensorFlow-XLA, and TensorFlow Lite) shows that W hite F ox can generate high-quality test programs to exercise deep optimizations requiring intricate conditions, practicing up to 8 times more optimizations than state-of-the-art fuzzers. To date, W hite F ox has found in total 101 bugs for the compilers under test, with 92 confirmed as previously unknown and 70 already fixed. Notably, W hite F ox has been recently acknowledged by the PyTorch team, and is in the process of being incorporated into its development workflow. Finally, beyond DL compilers, W hite F ox can also be adapted for compilers in different domains, such as LLVM, where WHiteFox has already found multiple bugs.Keywords
All Related Versions
This publication has 46 references indexed in Scilit:
- Effective test generation using pre-trained Large Language Models and mutation testingInformation and Software Technology, 2024
- Fuzzing Automatic Differentiation in Deep-Learning Libraries2023
- Evaluating and Improving Hybrid Fuzzing2023
- CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models2023
- Pangolin: Incremental Hybrid Fuzzing with Polyhedral Path Abstraction2020
- CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries2019
- Grey-Box Concolic Testing on Binary Code2019
- LLVM: A compilation framework for lifelong program analysis & transformation2004
- A generalization of the beta distribution with applicationsJournal of Econometrics, 1995
- ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLESBiometrika, 1933