The Empty Shelf, Audited: When Program Libraries Should Gate Synthesis

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Program libraries guide code generation, but a search may return no code or documentation relevant to the requested functionality. In this case, abstaining may exclude solvable tasks, whereas proceeding may waste computation on tasks likely to fail. Here, we introduce the synthesis abstention card, a verification framework for deciding whether missing library information justifies declining a task before generation. The protocol combines confidence intervals for support--failure linkage, retained risk, and calls avoided with checks of transport to the target workload. These checks classify decisions as supported, non-actionable, or uncertified against prespecified requirements. We examined these requirements through controlled studies and external audits. In a small, locally trained model, we found that authoritative implementation notes reduced the probability that all three generation attempts failed from 0.910 to 0.414. In a separate controlled system, abstention avoided 50.6% of generation calls when half the tasks lacked relevant library information. Abstention also reduced failure risk to 0.129 among retained tasks, compared with 0.366 when all tasks were attempted. These benefits persisted when 75% of tasks lacked library information, although retained risk was overestimated beyond the prespecified tolerance. The external audits revealed a further limitation: generic text signals could pass transport checks while providing little failure discrimination. Together, these findings show that library signals can justify abstention only when failure discrimination, retained risk, and savings are validated for the target workload. The card provides a basis for reassessing which tasks an agent is justified in attempting as its knowledge, capabilities, and workload change.