Random Testing and Evolutionary Testing for Fuzzing GraphQL APIs
Open Access
- 5 January 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on the Web
- Vol. 18 (1) , 1-41
Abstract
The Graph Query Language (GraphQL) is a powerful language for application programming interface (API) manipulation in web services. It has been recently introduced as an alternative solution for addressing the limitations of RESTful APIs. This article introduces an automated solution for GraphQL API testing. We present a full framework for automated API testing, from the schema extraction to test case generation. In addition, we consider two kinds of testing: white-box and black-box testing. The white-box testing is performed when the source code of the GraphQL API is available. Our approach is based on evolutionary search. Test cases are evolved to intelligently explore the solution space while maximizing code coverage and fault-finding criteria. The black-box testing does not require access to the source code of the GraphQL API. It is therefore of more general applicability, albeit it has worse performance. In this context, we use a random search to generate GraphQL data. The proposed framework is implemented and integrated into the open source EvoMaster tool. With enabled white-box heuristics (i.e., white-box mode), experiments on 7 open source GraphQL APIs and three search algorithms show statistically significant improvement of the evolutionary approach compared to the baseline random search. In addition, experiments on 31 online GraphQL APIs reveal the ability of the black-box mode to detect real faults.Keywords
Funding Information
- European Research Council (864972)
This publication has 59 references indexed in Scilit:
- Combining Multiple Coverage Criteria in Search-Based Unit Test Generation2015
- The Oracle Problem in Software Testing: A SurveyIEEE Transactions on Software Engineering, 2015
- Transactional and QoS-aware dynamic service composition based on ant colony optimizationFuture Generation Computer Systems, 2013
- A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineeringSoftware Testing, Verification and Reliability, 2012
- Search-based software engineeringACM Computing Surveys, 2012
- Testability transformationIEEE Transactions on Software Engineering, 2004
- A fast and elitist multiobjective genetic algorithm: NSGA-IIIEEE Transactions on Evolutionary Computation, 2002
- On the analysis of the (1+1) evolutionary algorithmTheoretical Computer Science, 2002
- No free lunch theorems for optimizationIEEE Transactions on Evolutionary Computation, 1997
- Automated software test data generationIEEE Transactions on Software Engineering, 1990