wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation
Authors:
- Benjamin HawksBHBenjamin Hawks
- Jason WeitzJWJason Weitz
- Dmitri DemlerDDDmitri Demler
- Karla Tame-NarvaezKTKarla Tame-Narvaez
- Dennis PlotnikovDPDennis Plotnikov
- Mohammad Mehdi RahimifarMRMohammad Mehdi Rahimifar
- Hamza Ezzaoui RahaliHRHamza Ezzaoui Rahali
- Audrey C. TherrienATAudrey C. Therrien
- Donovan SprouleDSDonovan Sproule
- Elham E. KhodaEKElham E. Khoda
- Keegan A. SmithKSKeegan A. Smith
- Russell MarroquinRMRussell Marroquin
Open Access
- 27 May 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Reconfigurable Technology and Systems
- Vol. 19 (2) , 1-29
Abstract
As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multiple efforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680,000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75th percentile within several percent of the synthesized resources on the synthetic test dataset.Keywords
Funding Information
- U.S. Department of Energy, Office of Science, Office of High Energy Physics (89243024CSC000002)
- U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS), Science Undergraduate Laboratory Internships Program
- National Science Foundation (CNS-1730158, ACI-1540112, ACI-1541349, OAC-1826967, OAC-2112167, CNS-2100237, CNS-2120019)
- Office of the President, University of California
- University of California San Diego, California Institute for Telecommunications and Information Technology / Qualcomm Institute
- National Science Foundation (2112356, 2411377)
- Fermi Research Alliance, LLC, U.S. Department of Energy (DOE), Office of Science, Office of High Energy Physics (DE-AC02-07CH11359)
- U.S. Department of Energy (DOE), Early Career Research Program (DE-0000247070)
- U.S. Department of Energy (KA2401045)
- U.S. Department of Energy (DOE), Office of Science, Office of Advanced Scientific Computing Research (DE-FOA-0002501, DE-SC0021396)
- Research Corporation for Science Advancement (CS-CSA-2023-109)
- Alfred P. Sloan Foundation (FG-2023-20452)
- U.S. Department of Energy (DOE), Office of Science, Office of High Energy Physics, Early Career Research Program (DE-SC0021187)
- National Science Foundation (NSF), Harnessing the Data Revolution (HDR) Institute for Accelerating AI Algorithms for Data Driven Discovery (PHY-2117997)
- U.S. Department of Energy (DOE), Office of High-Energy Physics (DE-SC-0023527)
- Canada Research Chairs (950-232805)
This publication has 19 references indexed in Scilit:
- rule4ml: an open-source tool for resource utilization and latency estimation for ML models on FPGAMachine Learning: Science and Technology, 2025
- Efficient compression at the edge for real-time data acquisition in a billion-pixel X-ray cameraNuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2023
- A Graph Neural Network Model for Fast and Accurate Quality of Result Estimation for High-Level SynthesisIEEE Access, 2023
- FlowGNN: A Dataflow Architecture for Real-Time Workload-Agnostic Graph Neural Network Inference2023
- Machine Learning Aided Hardware Resource Estimation for FPGA DNN Implementations2022
- Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectorsNature Machine Intelligence, 2021
- DB4HLS: A Database of High-Level Synthesis Design Space ExplorationsIEEE Embedded Systems Letters, 2021
- Fast inference of deep neural networks in FPGAs for particle physicsJournal of Instrumentation, 2018
- Fast and Accurate Estimation of Quality of Results in High-Level Synthesis with Machine Learning2018
- MachSuite: Benchmarks for accelerator design and customized architectures2014