Test Oracle Generation for REST APIs
Open Access
- 11 December 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (1) , 1-37
Abstract
The number and complexity of test case generation tools for REST APIs have significantly increased in recent years. These tools excel in automating input generation but are limited by their test oracles, which can only detect crashes, regressions, and violations of API specifications or design best practices. This article introduces AGORA+, an approach for generating test oracles for REST APIs through the detection of invariants—output properties that should always hold. AGORA+ learns the expected behavior of an API by analyzing API requests and their corresponding responses. We enhanced the Daikon tool for dynamic detection of likely invariants, adding new invariant types and creating a front-end called Beet. Beet translates any OpenAPI specification and a set of API requests and responses into Daikon inputs. AGORA+ can detect 106 different types of invariants in REST APIs. We also developed PostmanAssertify, which converts the invariants identified by AGORA+ into executable JavaScript assertions. AGORA+ achieved a precision of 80% on 25 operations from 20 industrial APIs. It also identified 48% of errors systematically seeded in the outputs of the APIs under test. AGORA+ uncovered 32 bugs in popular APIs, including Amadeus, Deutschebahn, GitHub, Marvel, NYTimesBooks, and YouTube, leading to fixes and documentation updates.Keywords
This publication has 75 references indexed in Scilit:
- Adaptive REST API Testing with Reinforcement Learning2023
- Improving Test Case Generation for REST APIs Through Hierarchical Clustering2021
- On testing machine learning programsJournal of Systems and Software, 2020
- RESTler: Stateful REST API Fuzzing2019
- Mutation Testing Advances: An Analysis and Survey2019
- ZoomIn: Discovering Failures by Detecting Wrong Assertions2015
- Practical automated vulnerability monitoring using program state invariants2013
- Regression testing minimization, selection and prioritization: a surveySoftware Testing, Verification and Reliability, 2012
- The Daikon system for dynamic detection of likely invariantsScience of Computer Programming, 2007
- Dynamically discovering likely program invariants to support program evolutionIEEE Transactions on Software Engineering, 2001