When to Say What: Learning to Find Condition-Message Inconsistencies
Authors:
- 1 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 868-880
Abstract
Programs often emit natural language messages, e.g., in logging statements or exceptions raised on unexpected paths. To be meaningful to users and developers, the message, i.e., what to say, must be consistent with the condition under which it gets triggered, i.e., when to say it. However, checking for inconsistencies between conditions and messages is challenging because the conditions are expressed in the logic of the programming language, while messages are informally expressed in natural language. This paper presents CMI-Finder, an approach for detecting condition-message inconsistencies. CMI-Finder is based on a neural model that takes a condition and a message as its input and then predicts whether the two are consistent. To address the problem of obtaining realistic, diverse, and large-scale training data, we present six techniques to generate large numbers of inconsistent examples to learn from automatically. Moreover, we describe and compare three neural models, which are based on binary classification, triplet loss, and fine-tuning, respectively. Our evaluation applies the approach to 300K condition-message statements extracted from 42 million lines of Python code. The best model achieves a precision of 78% at a recall of 72% on a dataset of past bug fixes. Applying the approach to the newest versions of popular open-source projects reveals 50 previously unknown bugs, 19 of which have been confirmed by the developers so far.Keywords
Funding Information
- European Research Council (851895)
This publication has 40 references indexed in Scilit:
- Code Prediction by Feeding Trees to Transformers2021
- IdBench: Evaluating Semantic Representations of Identifier Names in Source Code2021
- Detecting Error-Handling Bugs without Error Specification Input2019
- A Novel Neural Source Code Representation Based on Abstract Syntax Tree2019
- NL2Type: Inferring JavaScript Function Types from Natural Language Information2019
- Mutation Testing Advances: An Analysis and Survey2019
- Detecting fragile comments2017
- An exploratory study on exception handling bugs in Java programsJournal of Systems and Software, 2015
- Alattin: Mining Alternative Patterns for Detecting Neglected Conditions2009
- Mining exception-handling rules as sequence association rules2009