Do Programmers Prefer Predictable Expressions in Code?
Open Access
- 12 December 2020
- journal article
- research article
- Published by Wiley in Cognitive Science
- Vol. 44 (12) , e12921
Abstract
Source code is a form of human communication, albeit one where the information shared between the programmers reading and writing the code is constrained by the requirement that the code executes correctly. Programming languages are more syntactically constrained than natural languages, but they are also very expressive, allowing a great many different ways to express even very simple computations. Still, code written by developers is highly predictable, and many programming tools have taken advantage of this phenomenon, relying on language model surprisal as a guiding mechanism. While surprisal has been validated as a measure of cognitive load in natural language, its relation to human cognitive processes in code is still poorly understood. In this paper, we explore the relationship between surprisal and programmer preference at a small granularity—do programmers prefer more predictable expressions in code? Using meaning‐preserving transformations, we produce equivalent alternatives to developer‐written code expressions and run a corpus study on Java and Python projects. In general, language models rate the code expressions developers choose to write as more predictable than these transformed alternatives. Then, we perform two human subject studies asking participants to choose between two equivalent snippets of Java code with different surprisal scores (one original and transformed). We find that programmers do prefer more predictable variants, and that stronger language models like the transformer align more often and more consistently with these preferences.Keywords
Funding Information
- National Science Foundation (1414172)
This publication has 77 references indexed in Scilit:
- Random effects structure for confirmatory hypothesis testing: Keep it maximalJournal of Memory and Language, 2013
- Judy – a mutation testing tool for JavaIET Software, 2010
- Eye movement evidence that readers maintain and act on uncertainty about past linguistic inputProceedings of the National Academy of Sciences, 2009
- Data from eye-tracking corpora as evidence for theories of syntactic processing complexityCognition, 2008
- The Chicken or the Egg? A Probabilistic Analysis of English BinomialsLanguage, 2006
- Spoken syntax: The phonetics of giving a hand in New Zealand EnglishThe Linguistic Review, 2006
- Literate ProgrammingThe Computer Journal, 1984
- Syntactic/semantic interactions in programmer behavior: A model and experimental resultsInternational Journal of Parallel Programming, 1979
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960