Drishti Goel

Root cause analysis (RCA) for incidents in large-scale cloud systems is a complex, knowledge-intensive task that often requires significant manual effort from on-call engineers (OCEs). Improving RCA is vital for accelerating the incident resolution process and reducing service downtime and manual efforts. Recent advancements in Large-Language Models (LLMs) have proven to be effective in solving different stages of the incident management lifecycle including RCA. However, existing LLM-based RCA recommendations typically leverage default finetuning or retrieval augmented generation (RAG) methods with static, manually designed prompts, which lead to sub-optimal recommendations. In this work, we leverage 'PromptWizard', a state-of-the-art prompt optimization technique, to automatically identify the best optimized prompt instruction that is combined with semantically similar historical examples for querying underlying LLMs during inference. Moreover, by utilizing more than 180K historical incident data from Microsoft, we developed cost-effective finetuned small language models (SLMs) for RCA recommendation generation and demonstrate the power of prompt optimization on such domain-adapted models. Our extensive experimental results show that prompt optimization can improve the accuracy of RCA recommendations by 21% and 13% on 3K test incidents over RAG-based LLMs and finetuned SLMs, respectively. Lastly, our human evaluation with incident owners have demonstrated the efficacy of prompt optimization on RCA recommendation tasks. These findings underscore the advantages of incorporating prompt optimization into AI for Operations (AIOps) systems, delivering substantial gains without increasing computational overhead.

Trust and reliance are often treated as coupled constructs in human-AI interaction research, with the assumption that calibrating trust will lead to appropriate reliance. We challenge this assumption in educational contexts, where students increasingly turn to AI for learning support. Through semi-structured interviews with graduate students (N=8) comparing AI-generated and human-generated responses, we find a systematic dissociation: students exhibit high trust but low reliance on human experts due to social barriers (fear of judgment, help-seeking anxiety), while showing low trust but high reliance on AI systems due to social affordances (accessibility, anonymity, judgment-free interaction). Using Mutual Theory of Mind as an analytical lens, we demonstrate that trust is shaped by epistemic evaluations while reliance is driven by social factors—and these may operate independently.

Incident management for large cloud services is a complex and tedious process that requires a significant amount of manual effort from on-call engineers (OCEs). OCEs typically leverage data from different stages of the software development lifecycle [SDLC] (e.g., codes, configuration, monitor data, service properties, service dependencies, trouble-shooting documents, etc.) to generate insights for detection, root cause analysis and mitigation of incidents. Recent advancements in large language models [LLMs] (e.g., ChatGPT, GPT-4, Gemini) have created opportunities to automatically generate contextual recommendations for the OCEs, assisting them in quickly identifying and mitigating critical issues. However, existing research typically takes a silo-ed view of solving a certain task in incident management by leveraging data from a single stage of the SDLC. In this paper, we demonstrate that augmenting additional contextual data from different stages of the SDLC improves the performance of two critically important and practically challenging tasks: (1) automatically generating root cause recommendations for dependency failure related incidents, and (2) identifying the ontology of service monitors used for automatically detecting incidents. By leveraging a dataset of 353 incidents and 260 monitors from Microsoft, we demonstrate that augmenting contextual information from different stages of the SDLC improves the performance over state-of-the-art methods.