Statically Contextualizing Large Language Models with Typed Holes
Open Access
- 8 October 2024
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Programming Languages
- Vol. 8 (OOPSLA2) , 468-498
Abstract
Large language models (LLMs) have reshaped the landscape of program synthesis. However, contemporary LLM-based code completion systems often hallucinate broken code because they lack appropriate code context, particularly when working with definitions that are neither in the training data nor near the cursor. This paper demonstrates that tighter integration with the type and binding structure of the programming language in use, as exposed by its language server, can help address this contextualization problem in a token-efficient manner. In short, we contend that AIs need IDEs, too! In particular, we integrate LLM code generation into the Hazel live program sketching environment. The Hazel Language Server is able to identify the type and typing context of the hole that the programmer is filling, with Hazel’s total syntax and type error correction ensuring that a meaningful program sketch is available whenever the developer requests a completion. This allows the system to prompt the LLM with codebase-wide contextual information that is not lexically local to the cursor, nor necessarily in the same file, but that is likely to be semantically local to the developer’s goal. Completions synthesized by the LLM are then iteratively refined via further dialog with the language server, which provides error localization and error messages. To evaluate these techniques, we introduce MVUBench, a dataset of model-view-update (MVU) web applications with accompanying unit tests that have been written from scratch to avoid data contamination, and that can easily be ported to new languages because they do not have large external library dependencies. These applications serve as challenge problems due to their extensive reliance on application-specific data structures. Through an ablation study, we examine the impact of contextualization with type definitions, function headers, and errors messages, individually and in combination. We find that contextualization with type definitions is particularly impactful. After introducing our ideas in the context of Hazel, a low-resource language, we duplicate our techniques and port MVUBench to TypeScript in order to validate the applicability of these methods to higher-resource mainstream languages. Finally, we outline ChatLSP, a conservative extension to the Language Server Protocol (LSP) that language servers can implement to expose capabilities that AI code completion systems of various designs can use to incorporate static context when generating prompts for an LLM.Keywords
All Related Versions
Funding Information
- National Science Foundation (2238744)
This publication has 41 references indexed in Scilit:
- Gradual Structure Editing with Obligations2023
- What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?2023
- Examining Zero-Shot Vulnerability Repair with Large Language Models2023
- Towards More Effective AI-Assisted Programming: A Systematic Design Exploration to Improve Visual Studio IntelliCode’s User Experience2023
- Program sketching with live bidirectional evaluationProceedings of the ACM on Programming Languages, 2020
- Merlin: a language server for OCaml (experience report)Proceedings of the ACM on Programming Languages, 2018
- On the naturalness of softwareCommunications of the ACM, 2016
- OverCodeACM Transactions on Computer-Human Interaction, 2015
- Defects4J: a database of existing faults to enable controlled testing studies for Java programs2014
- Contextual modal type theoryACM Transactions on Computational Logic, 2008