Comfrey: Mitigating Integration Failures in LLM-enabled Software at Run-Time
Open Access
- 12 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3014-3027
Abstract
Due to the unrestricted outputs of LLMs and strict requirements of software components, integration failures are widespread in software that incorporates LLM agents and retrieval-augmented generation (RAG). Even seemingly correct LLM/RAG responses can trigger software misbehaviors if they violate these requirements. In this paper, we conduct an empirical study to understand integration failures in real-world LLM-enabled applications. Guided by this study, we present Comfrey [104], a runtime framework that adapts the LLM agent and RAG responses to meet software requirements, serving as a middle layer between AI and software components. It automatically detects and resolves potential integration failures through a three-stage workflow, ensuring component compatibility. Our evaluation with a variety of open-source applications demonstrates that Comfrey detects 75.1% and prevents 63.3% of potential integration failures with 8.4% overhead, significantly outperforming the baselines.Keywords
This publication has 73 references indexed in Scilit:
- A study on prompt design, advantages and limitations of ChatGPT for deep learning program repairAutomated Software Engineering, 2025
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation2024
- RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback2024
- MaPLe: Multi-modal Prompt Learning2023
- An Empirical Study of Deep Learning Models for Vulnerability Detection2023
- Exploratory Data Analysis2022
- A Replication of Are Machine Learning Cloud APIs Used Correctly2021
- Tackling Build Failures in Continuous Integration2019
- Apricot: A Weight-Adaptation Approach to Fixing Deep Learning Models2019
- Automated analysis of free speech predicts psychosis onset in high-risk youthsSchizophrenia, 2015