Chetan Bansal

Agentic applications are LLMs that iteratively invoke external tools to accomplish complex tasks. Such tool-based agents are rapidly becoming the dominant paradigm for deploying language models in production. Unlike traditional single-turn inference, agentic workloads chain together multiple LLM calls and tool executions before producing a final response, creating a new performance bottleneck that manifests as increased latency in First Token Rendered (FTR) of the final answer. Through analysis of requests at production scale, we reveal three critical challenges: tool calls account for 30-85% of FTR latency, KV cache hit rates collapse despite substantial context reuse across iterations, and sequential orchestration wastes potential intra-request parallelism. These bottlenecks stem from a design gap in which orchestrators and LLM engines operate as decoupled black boxes, preventing cross-layer optimizations. We present Sutradhara, a co-designed agentic inference system that integrates orchestration with LLM serving through a thin API enabling three optimizations: overlap tool execution with subsequent LLM prefill using tool-aware prompt splitting, streaming tool execution to dispatch tools incrementally during decode rather than waiting for complete output, and orchestrator-aware cache management that uses semantic hints to improve hit rates and reduce thrashing. Implemented on vLLM, Sutradhara improves the throughput–latency trade-off in agentic systems, sustains up to 77% higher load at the same median FTR latency, or reduces median FTR latency by up to 15% at the same load while reducing end-to-end latency by up to 11% on A100 GPUs.

Large language models (LLMs) have shown growing potential in software engineering, yet few benchmarks evaluate their ability to repair software during migration across instruction set architectures (ISAs). Cross-ISA migration, such as between x86_64 and aarch64 , requires handling complex dependencies, heterogeneous toolchains, and long build logs while ensuring executable verification. To address this challenge, we present Build-bench (Homepage: https://aiops-lab-nku.github.io/Build-bench/ , Code at https://github.com/zcyyc/Build-bench ), an end-to-end benchmark that systematically evaluates the capability of LLMs to repair build failures in cross-ISA settings. Build-bench collects 268 real-world failed packages and integrates auxiliary tools including Structure Extraction , File Content Extraction , Content Modification , and Build Verification to support autonomous, tool-augmented reasoning. The repair process operates in an iterative loop where, upon failure, the model receives updated build logs and previous repair outcomes to refine subsequent attempts. Through a comparative evaluation across the studied models, Build-bench reveals that current models achieve a maximum build success rate of 63.19% and tool usage patterns differ significantly across models. By coupling real build environments with verifiable outcomes, Build-bench establishes the first architecture-aware benchmark for studying LLM-based software build and repair.

Root cause analysis (RCA) for incidents in large-scale cloud systems is a complex, knowledge-intensive task that often requires significant manual effort from on-call engineers (OCEs). Improving RCA is vital for accelerating the incident resolution process and reducing service downtime and manual efforts. Recent advancements in Large-Language Models (LLMs) have proven to be effective in solving different stages of the incident management lifecycle including RCA. However, existing LLM-based RCA recommendations typically leverage default finetuning or retrieval augmented generation (RAG) methods with static, manually designed prompts, which lead to sub-optimal recommendations. In this work, we leverage 'PromptWizard', a state-of-the-art prompt optimization technique, to automatically identify the best optimized prompt instruction that is combined with semantically similar historical examples for querying underlying LLMs during inference. Moreover, by utilizing more than 180K historical incident data from Microsoft, we developed cost-effective finetuned small language models (SLMs) for RCA recommendation generation and demonstrate the power of prompt optimization on such domain-adapted models. Our extensive experimental results show that prompt optimization can improve the accuracy of RCA recommendations by 21% and 13% on 3K test incidents over RAG-based LLMs and finetuned SLMs, respectively. Lastly, our human evaluation with incident owners have demonstrated the efficacy of prompt optimization on RCA recommendation tasks. These findings underscore the advantages of incorporating prompt optimization into AI for Operations (AIOps) systems, delivering substantial gains without increasing computational overhead.

Major cloud outages require responders to make high-stakes decisions from noisy, multimodal, and continuously evolving operational data. While recent AI-assisted approaches help with tasks such as triage, troubleshooting, and root-cause analysis, they provide limited support for lifecycle-wide outage management in production environments. We present ActionNex, a production-grade, memory-augmented system that recommends state-aware next-best-actions throughout the outage lifecycle. ActionNex converts fragmented multimodal signals into structured operational context and grounds recommendation in three complementary memories: long-term operational knowledge encoded as Key–Condition–Action (KCA) units, episodic memory from historical outage cases, and working memory derived from the current outage state. We evaluate ActionNex on eight real production outages totaling approximately 9 million tokens of multimodal operational data. Using two held-out outages and six incremental learning episodes, we find that action-level recall improves consistently as episodic memory accumulates, rising from 73.8% to 79.6% on one outage and from 65.1% to 73.5% on the other, while precision remains broadly stable. Additional analyses show that retrieval strategy provides a practical precision–recall control knob, and that structured memory outperforms prompt-only alternatives while reducing over-generation. A 30-day pilot across 10 services provides early evidence of usefulness under live operational conditions. Together, these results indicate that memory-grounded, human-in-the-loop recommendation is a practical and effective design for production outage management.

Modern cloud services increasingly rely on agentic software operations, where AI agents, rule-based automation (monitoring alerts, health checks, template runbooks), and human engineers coordinate incident response through shared discussion threads. As these systems mature, the on-call engineer faces a growing volume of machine-generated messages and must quickly identify which ones actually help, under minutes-level time pressure where every moment spent reading noise delays mitigation. We study this problem across thousands of production incidents from 5 cloud services over four years (2022 to 2026). We define a good first message (GFM): the first machine-generated message in an incident discussion thread whose specific content demonstrably influences the on-call engineer’s next investigation action. Three findings emerge. First, incident threads grew 7× longer (from a median of 4 to 30 messages) as AI agents were introduced, yet noise (messages lacking substantive technical content) decreased: among the three actor types, AI agents, rule-based automation, and humans, AI produces the lowest noise rate. Second, AI agents and automation are complementary, covering different workflow roles (e.g., diagnosis vs. monitoring) rather than duplicating each other. Third, only about one in five incidents contains a GFM; influence varies dramatically by functional role but far less by actor type: what a message says matters more than who posted it. We further show that GFMs can be surfaced automatically: σ-rank, an LLM ranker whose prompt encodes domain knowledge from the empirical study, places the GFM in the top 3 of ∼20 machine messages 89% of the time, reducing the engineer’s noise-reading burden by 7×. These results provide the first empirical evidence of how deployed AI output is actually consumed during live incidents and offer a practical mechanism to cut through information overload in time-critical incident response.

Frequent toolchain updates and the expanding diversity of instruction set architectures (ISAs) have made large-scale system-level software package repair a critical task. Diagnosing and repairing build failures remains challenging due to heterogeneous failure evidence, complex dependency constraints, and architecture-specific build conventions. While recent LLM-based repair methods have shown promise for project-level source code fixes, they struggle with system-level repair where failures involve multi-language artifacts (e.g., build recipes, scripts, and source archives) and require iterative validation through external build services. In this paper, we first conduct a systematic empirical study of real-world system-level build failures. Our findings reveal that 72% of successful repairs primarily involve adjustments to build configurations, dependencies, or environment settings rather than isolated source-code modifications, suggesting that effective repair must prioritize packaging logic and iterative feedback. Motivated by these insights, we propose EvidenT, an evidence-preserving repair framework that decouples iteration-aware evidence management from tool execution. EvidenT comprises (1) an external Build Service for reproducible build execution and feedback; (2) an Evidence-Preserving Repair Controller that performs cross-modal fusion of repair history, knowledge context, and build artifacts; and (3) an automated Repair Orchestrator that executes a suite of modular tools for failure localization and system-level repair actions within a closed-loop validation environment. We evaluate EvidenT on a benchmark of 219 real-world RISC-V package build failures. EvidenT successfully repairs 118 packages (53.88%), substantially outperforming state-of-the-art agentic baselines (20.55%) and direct LLM-based repair (1.83%). To demonstrate its architectural generality, we extend EvidenT to other ISAs by updating only ISA-specific knowledge context. In preliminary experiments, it achieves success rates of 41.77% on aarch64 and 46.99% on x86_64, showcasing its robustness across diverse hardware ecosystems.

Global cloud service providers handle inference workloads for LLMs that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks, with diverse and often conflicting Service Level Agreement (SLA) needs. Serving such mixed workloads is challenging due to the multiple models, diverse GPU hardware, and globally distributed data centers. Solutions that silo fast and slow tasks on separate GPU pools cause under-utilization of expensive accelerators. Here, we characterize the large-scale LLM inference workloads (> 10 M requests per day) at Microsoft Office 365, and highlight key observations in one of the first Internet-scale public studies. These insights drive SageServe , an LLM serving framework that dynamically adapts to workload demands. It combines short-term request routing to data centers with long-term scaling of GPU VMs and model placement with higher lead times, and co-optimizes the routing and resource allocation problem using a traffic forecast model and an ILP solution. We evaluate SageServe on traces comprising 10M production requests across three regions and four open-source models. SageServe achieves up to 25% savings in GPU-hours compared to the current baseline deployment and reduces GPU-hour wastage due to inefficient auto-scaling by 80%, potentially resulting in monthly cost savings of up to $2.5 million, while maintaining tail latency and meeting SLAs.

Maintaining up-to-date troubleshooting guides (TSGs) is critical for the reliability of cloud systems, yet manual maintenance often leads to inefficiencies and outdated documentation. This paper proposes TSGen, an automated pipeline for generating high-quality, structured TSGs from historical incident reports using large language models (LLMs). Our approach consists of three stages: (1) filtering and classifying incident data into diagnostically relevant categories, (2) distilling core incidents to ensure diversity and generalizability, and (3) organizing the distilled knowledge into a directed acyclic graph (DAG) that captures root causes and resolutions in a structured manner. By leveraging real-world incident discussions, TSGen produces dynamic and reusable guides tailored for live troubleshooting. Experiments on real-world incidents from Microsoft demonstrate that TSGen achieves 54.8% incident coverage and approximately 3× higher retrieval accuracy compared to baselines. Furthermore, the system supports iterative updates, allowing guides to evolve alongside dynamic cloud environments. Human evaluation shows that on-call engineers rate these generated TSGs significantly higher than human-crafted ones.