Xiwen Teoh

Large Language Models (LLMs) have significantly improved programming efficiency by parsing natural language into code snippets. However, their performance degrades significantly as requirements scale; when faced with multi-modal documents containing hundreds of scenarios, LLMs often produce incorrect implementations or omit crucial constraints. Observing LLMs' ever-evolving capability and their persistent stochastic hallucination, we raise a question: whether it is possible to make LLM-based agentic programming go beyond "code generation" to "requirement compilation", i.e., whether programmers can produce a runnable system by only accomplishing (non-trivial) requirement documents? In this work, we take a first step by proposing the ARC (Agentic Requirement Compilation) technique to parse a multi-modal requirement document, describing hundreds of scenarios in a DSL format, into a runnable software system. In addition to the source code, ARC also generates software engineering artifacts including (1) a modular design that spans the user interface, API interface, and database, (2) enriched test cases for each interface (including unit tests, modular tests, and integration tests), and (3) detailed traceability across all artifacts for software maintenance. Our approach employs a bidirectional test-driven agentic loop: (1) a top-down architecture phase that decomposes requirements into UI, API, and database interfaces, each of which is equipped with verifiable test suites, and (2) a bottom-up implementation phase where agents generate code that must satisfy the generated tests. Throughout this process, ARC maintains strict traceability across requirements, design, and code to facilitate intelligent asset reuse and follow-up maintenance. We evaluate ARC on two complementary benchmarks, i.e., a depth-oriented benchmark of 6 runnable web systems spanning 50-200 requirement scenarios, and the breadth-oriented AppForge benchmark comprising 101 Android app generation tasks. Across 3 independent trials, ARC outperforms all state-of-the-art LLM-based baselines, with the generated web systems passing on average 50.6% more GUI tests, and achieving 100% compile success and 68.3% test case pass rate on AppForge. In addition, a user study with 21 participants shows that participants with limited programming experience successfully write DSL-based documents consisting of 50 to 174 scenarios, within 5.6 hours on average, to generate a runnable system such as a real-world ticket-booking system of around 10K lines of code with maintainable architecture.

Visual language model (VLM) agents show great promise in automating graphical user interface (GUI) testing against requirements in natural language. However, the probabilistic nature of language models can have inherent hallucinations. Therefore, given a detected inconsistency between the requirement and the web application, it is hard to distinguish whether it stems from the hallucination or a real application bug. Addressing this issue presents two core technical challenges: (1) limited capability and accuracy in deriving implicit test oracles, where the agent must act as its own oracle to implicitly decide if the application’s behavior is correct without guidance, and (2) limited reliability due to probabilistic inference, where an LLM’s inconsistent reasoning undermines its trustworthiness as an oracle. We introduce WebTestPilot, a neurosymbolic LLM-based approach that addresses both challenges through symbolization. WebTestPilot detects and abstracts critical GUI elements of a web application into symbolic variables. This design improves reliability by constraining assertion generation to operations grounded in explicitly defined symbols, thereby reducing unconstrained or inconsistent reasoning. At the same time, it improves accuracy by representing application states and their relationships in a structured symbolic form, which increases the likelihood of the agent recognizing data, causal, and temporal dependencies across states. Together, these capabilities enable WebTestPilot to generate reliable and accurate test oracles that capture meaningful implicit expectations derived from test requirements. To advance research in this area, we build a benchmark of bug-injected web apps for evaluating NL-to-E2E testing. The results show that WebTestPilot achieves a task completion rate of 99%, with 96% precision and 96% recall in bug detection, outperforming the best baseline (+70 precision, +27 recall). The agent generalizes across diverse natural language inputs (i.e., those containing typos, grammatical errors, redundant sentences, stylistic restyling, or abbreviations) and model scales (3B–72B). In a real-world deployment with a no-code platform, WebTestPilot discovered 8 bugs during development, including data binding, UI, and navigation issues.

GUI testing is crucial for ensuring the reliability of mobile applications. State-of-the-art GUI testing approaches are successful in exploring more application scenarios and discovering general bugs such as application crashes. However, industrial GUI testing also needs to investigate application-specific bugs such as deviations in screen layout, widget position, or GUI transition from the GUI design mock-ups created by the application designers. These mock-ups specify the expected screens, widgets, and their respective behaviors. Validating the consistency between the GUI design and the implementation is labor-intensive and time-consuming, yet, this validation step plays an important role in industrial GUI testing. In this work, we propose , an approach for detecting inconsistencies between the mobile design and their implementations. The mobile design usually consists of design mock-ups that specify (1) the expected screen appearances (e.g., widget layouts, colors, and shapes) and (2) the expected screen behaviors, regarding how one screen can transition into another (e.g., labeled widgets with textual description). Given a design mock-up and the implementation of its application, reports both their screen inconsistencies as well as process inconsistencies. On the one hand, detects the screen inconsistencies by abstracting every screen into a widget container where each widget is represented by its position, width, height, and type. By defining the partial order of widgets and the costs of replacing, inserting, and deleting widgets in a screen, we convert the screen-matching problem into an optimizable widget alignment problem. On the other hand, we translate the specified GUI transition into stepwise actions on the mobile screen (e.g., click, long-press, input text on some widgets). To this end, we propose a visual prompt for the vision-language model to infer widget-specific actions on the screen. By this means, we can validate the presence or absence of expected transitions in the implementation. Our extensive experiments on 80 mobile applications and 160 design mock-ups show that (1) can achieve 99.8% precision and 98.6% recall in detecting screen inconsistencies, outperforming the state-of-the-art approach, such as GVT, by 66.2% and 56.6% respectively, and (2) reports zero errors in detecting process inconsistencies. Furthermore, our industrial case study on applying on a trading mobile application shows that has detected nine application bugs, and all the bugs were confirmed by the original application experts. Our code is available at https://github.com/code-philia/GUIPilot.

Web applications are crucial infrastructures in the modern society, which have high demand of reliability and security. However, their frontend can be manipulable by the clients (e.g., the frontend code can be modified to bypass some validation steps), which incurs the runtime anomaly when operating the web service. Existing state-of-the-art anomaly detectors largely learn a deep learning model from the collected logs to predict abnormal logs with a probability. While effective in general, those approaches can suffer from (1) inaccuracy caused by subtle difference between the normal and abnormal/attack logs and (2) additional efforts for root cause analysis. In this work, we propose WebNorm, an anomaly detection approach to detect and explain the attack-caused anomalies on web applications in a unified way. Our rationale lies in learning the behaviorial normalities of a running web application as invariants. The normalities are designed regarding data normality (e.g., what information must be consistent across different events), flow normality (e.g., what events must happen under certain circumstances), and common-sense normality (e.g., what is the normal range of some parameters). The violation of the invariants indicates both the alarm and its explanation. WebNorm first monitors the normal behaviors of subject application and captures its information flows between entities such as frontend, service, and database. Then, it learns the behaviorial normalities in terms of logical rules so that it can detect and explain behaviorial anomaly by the inconsistency between the learned normalities and the runtime application behaviors. We model the invariants as first-order logics, transferrable to executable Python scripts to generate alarm with explainable root cause. Our extensive experiment shows that, on detecting the tamper attacks on the web applications as TrainTicket and NiceFish. WebNorm improves the precision and the recall of the baselines such as LogAnomaly, LogRobust, DeepLog, NeuralLog, PLELog, ReplicaWatcher by more than 56.1% and 35.1% respectively, serving as a new state-of-the-art anomaly detection solution.