When AI Writes Code: Investigating Security Issues in Agentic Software Changes
Open Access
- 13 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 832-836
Abstract
AI code agents are increasingly used to generate software changes, raising questions about whether their outputs can be trusted. We investigate this by quantifying security issues in agent-generated code using a repository-level Static Application Security Testing (SAST) pipeline. We analyze 7,200 Agent Pull Requests (APRs) and 6,620 Human Pull Requests (HPRs) from 818 repositories with at least 500 GitHub stars, using four SAST engines to assess the introduction and severity of Common Weakness Enumeration (CWE) indicators. Our results show that agents introduce security risks (RQ1), including critical weaknesses such as hard-coded credentials and command injection. However, security performance varies across agents and task types, with statistically significant differences emerging for specific tasks (RQ2). Comparing agents to humans (RQ3), we find that agents introduce fewer or less severe issues on small, focused bug fixes, while differences on larger pull requests are task-dependent and driven by a small number of high-impact changes rather than systematic performance gaps. Overall, agent-generated code is neither uniformly secure nor insecure; its safety depends on task context, emphasizing the need for task-aware deployment and human-oversight.Keywords
This publication has 11 references indexed in Scilit:
- Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories2025
- Towards Explainable Vulnerability Detection With Large Language ModelsIEEE Transactions on Software Engineering, 2025
- Semgrep*: Improving the Limited Performance of Static Application Security Testing (SAST) Tools2024
- Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers2024
- On the Prevalence, Co-occurrence, and Impact of Infrastructure-as-Code Smells2024
- Lifting On-Demand Analysis to Higher-Order Languages2023
- Guidelines for the search strategy to update systematic literature reviews in software engineeringInformation and Software Technology, 2020
- Architectural technical debt identification2018
- The promises and perils of mining GitHub2014
- A practical guide for using statistical tests to assess randomized algorithms in software engineering2011