Setup AGent (SAG): A Dual-Model LLM Agent for Autonomous End-to-End Java Project Configuration
Open Access
- 12 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 96-100
Abstract
Environment setup remains a critical bottleneck in software development, with Java projects presenting unique challenges due to complex build tool configurations, transitive dependencies, and compilation requirements. While existing solutions like Installamatic achieve 55% success on Python repositories, Java/JVM repositories show only 29.47% success rates with current approaches. We present SAG (Setup Agent), an LLM-based system specifically designed for Java project environment setup within Docker containers. SAG employs three key innovations: a dual-model ReAct architecture separating reasoning from action, hierarchical context management with a "Trunk/Branch" context system for complex task chains, and independent file validation ensuring objective assessment while maintaining autonomous discovery. Through evaluation on 15 Apache projects across four model configurations, SAG achieves 78.9% overall build success and 48.4% test execution rate. The o4-mini+GPT-4.1-mini configuration achieves the highest 84.4% success using only 17,956 tokens, compared to GPT-5+GPT-5’s 75.6% success with 99,400 tokens—a 5.5× reduction in computational resources with 8.8 percentage points higher success. The extreme 12.09 thinking-to-action token ratio validates our architectural separation. Our results establish that role specialization and focused model pairing outperform uniformly applying powerful models, paving the way for efficient, task-specific automation solutions.Keywords
This publication has 10 references indexed in Scilit:
- Beyond pip Install: Evaluating LLM Agents for the Automated Installation of Python Projects2025
- Deep Learning for Code Intelligence: Survey, Benchmark and ToolkitACM Computing Surveys, 2024
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering2024
- SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories2024
- Large Language Models for Software Engineering: Survey and Open Problems2023
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation2023
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation2023
- Software documentation2020
- Moving into a new software project landscape2010
- A Survey of Documentation Practice within Corrective MaintenanceEmpirical Software Engineering, 2005