PSB2
Authors:
Open Access
- 26 June 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 785-794
Abstract
For the past six years, researchers in genetic programming and other program synthesis disciplines have used the General Program Synthesis Benchmark Suite to benchmark many aspects of automatic program synthesis systems. These problems have been used to make notable progress toward the goal of general program synthesis: automatically creating the types of software that human programmers code. Many of the systems that have attempted the problems in the original benchmark suite have used it to demonstrate performance improvements granted through new techniques. Over time, the suite has gradually become outdated, hindering the accurate measurement of further improvements. The field needs a new set of more difficult benchmark problems to move beyond what was previously possible. In this paper, we describe the 25 new general program synthesis benchmark problems that make up PSB2, a new benchmark suite. These problems are curated from a variety of sources, including programming katas and college courses. We selected these problems to be more difficult than those in the original suite, and give results using PushGP showing this increase in difficulty. These new problems give plenty of room for improvement, pointing the way for the next six or more years of general program synthesis research.Keywords
This publication has 32 references indexed in Scilit:
- A Grammar Design Pattern for Arbitrary Program Synthesis Problems in Genetic Programming2017
- Field Report: Applying Monte Carlo Tree Search for Program Synthesis2016
- General Program Synthesis Benchmark Suite2015
- Solving Uncompromising Problems With Lexicase SelectionIEEE Transactions on Evolutionary Computation, 2014
- Benchmarks that matter for genetic programming2014
- Better GP benchmarks: community survey results and proposalsGenetic Programming and Evolvable Machines, 2012
- Genetic programming needs better benchmarks2012
- Automating string processing in spreadsheets using input-output examplesACM SIGPLAN Notices, 2011
- Recent Improvements of MagicHaskeller2010
- The Push3 execution stack and the evolution of control2005