Jose Antonio Zamudio Amaya

Even in the age of AI, generating comprehensive test inputs for software systems remains a challenge, particularly for interactive and reactive systems, which require entire interaction sequences to reach the desired states or to cover missing behavior. While modern fuzzers are effective at generating random inputs that test the robustness of input processors, they still struggle to produce or mutate complex inputs and interactions. LLM-based systems, on the other hand, can generate small example inputs, but fail to systematically explore the space of inputs and interactions as would be required for comprehensive testing. In this tutorial, we introduce test generation with fandango, a modern specification-based generator of test inputs and interactions. Based on a spec file that defines the structure and properties of the program input, fandango produces inputs that are syntactically and semantically correct, and systematically cover the input space of the program under test. In three interactive sessions, we (1) introduce language-based testing, (2) demonstrate how to use constraints to define desired properties, and then (3) delve into full-fledged protocol testing, producing entire interaction sequences between fandango and the network components under test. Participants are expected to bring a basic understanding of software testing; knowledge of python is a plus, but not required. At the end of the tutorial, they will (1) be able to specify and test formats for inputs and interactions, (2) produce comprehensive test suites using language-based techniques, and (3) understand the trade-offs and practical applications of language-based testing.

Language-based fuzzers leverage formal input specifications (languages) to generate arbitrarily large and diverse sets of valid inputs for a program under test. Modern language-based test generators combine grammars and constraints to satisfy syntactic and semantic input constraints. ISLA, the leading input generator in that space, uses symbolic constraint solving to solve input constraints. Using solvers places ISLA among the most precise fuzzers but also makes it slow. In this paper, we explore search-based testing as an alternative to symbolic constraint solving. We employ a genetic algorithm that iteratively generates candidate inputs from an input specification, evaluates them against defined constraints, evolving a population of inputs through syntactically valid mutations and retaining those with superior fitness until the semantic input constraints are met. This evolutionary procedure, analogous to natural genetic evolution, leads to progressively improved inputs that cover both semantics and syntax. This change boosts the efficiency of language-based testing: In our experiments, compared to ISLA, our search-based FANDANGO prototype is faster by one to three orders of magnitude without sacrificing precision. The search-based approach no longer restricts constraints to constraint solvers' (miniature) languages. In FANDANGO, constraints can use the whole Python language and library. This expressiveness gives testers unprecedented flexibility in shaping test inputs. It allows them to state arbitrary goals for test generation: ''Please produce 1,000 valid test inputs where the voltage field follows a Gaussian distribution but never exceeds 20 mV.''

Fuzzing is an essential method for finding vulnerabilities. Conventional fuzzing looks across a wide input space, but it cannot handle systems that need intricate and specialized input patterns. Grammar-based fuzzing uses formal grammars to shape the inputs the fuzzer generates. This method is crucial for directing fuzzers to generate complicated inputs that adhere to syntactical requirements. However, existing approaches are biased towards certain input features, leading to significant portions of the solution space being under-explored or ignored. In this paper, we review the state-of-the-art methods, emphasizing the limitations of grammar-based fuzzing, and we provide a first approach for incorporating distribution sampling into fuzzing, accompanied by encouraging first findings. This work can represent a significant step towards achieving comprehensive input space exploration in grammar-based fuzzing, with implications for enhancing the robustness and reliability of the fuzzing targets.

The development and maintenance of feature models is often an error-prone activity requiring different types of analysis operations that help developers to restore required feature model properties. Fulfilling such properties helps to assure compliance between feature model and corresponding domain variability properties and -at the same time - helps to increase feature model maintainability. In this paper, we propose a set of additional analysis operations that provide insights regarding potential impacts of applying feature models in constraint-based recommendation scenarios where feature models are used to define user preference spaces. Our proposed analysis operations provide a.o. insights into aspects such as feature restrictiveness and product accessibility when applying a constraint-based recommender system. We analyze usage scenarios of the operations on the basis of an example implementation with a digital camera feature model and discuss open research issues.