Process-Centric Analysis of Agentic Software Systems
Authors:
- Shuyang LiuSLShuyang Liu
- Yang ChenYCYang Chen
- Rahul KrishnaRKRahul Krishna
- Saurabh SinhaSSSaurabh Sinha
- Jatin GanhotraJGJatin Ganhotra
- Reyhan JabbarvandRJReyhan Jabbarvand
Preprint
Abstract
Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their execution (i.e., trajectories) is inherently stochastic and adaptive to the problem they are solving. Evaluation of such systems is often outcome-centric, judging their performance based on success or failure at the final step. This narrow focus overlooks detailed insights about such systems, failing to explain how agents reason, plan, act, or change their strategies over time. Inspired by the structured representation of conventional software systems as graphs, we introduce Graphectory to systematically encode the temporal and semantic relations in such software systems. Graphectory facilitates the design of process-centric metrics and analyses to assess the quality of agentic workflows independent of final success. Using Graphectory, we analyze 4000 trajectories of two dominant agentic programming workflows, namely SWE-agent and OpenHands, with a combination of four backbone Large Language Models (LLMs), attempting to resolve SWE-bench Verified issues. Our fully automated analyses reveal that: (1) agents using richer prompts or stronger LLMs exhibit more complex Graphectory, reflecting deeper exploration, broader context gathering, and more thorough validation before patch submission; (2) agents' problem-solving strategies vary with both problem difficulty and the underlying LLM -- for resolved issues, the strategies often follow coherent localization-patching-validation steps, while unresolved ones exhibit chaotic, repetitive, or backtracking behaviors; (3) even when successful, agentic programming systems often display inefficient processes, leading to unnecessarily prolonged trajectories.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Demystifying LLM-Based Software Engineering AgentsProceedings of the ACM on Software Engineering, 2025
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution2024
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models2023
- Reflexion: language agents with verbal reinforcement learning2023
- MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting2023
- GenProg: A Generic Method for Automatic Software RepairIEEE Transactions on Software Engineering, 2011
- Mining periodic-frequent patterns with maximum items' support constraints2010
- A Complexity MeasureIEEE Transactions on Software Engineering, 1976