Casey Casalnuovo
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Casey Casalnuovo has a total of 17 co-authors. Here are the top co-authors by number of shared publications:
Livermore Sandia National Laboratories (Snl-Ca)
7 shared publicationsUsdoe National Nuclear Security Administration (Nnsa)
7 shared publicationsDavid Minh Kavaler
Sandia National Laboratories (SNL-CA), Livermore, CA (United States)
6 shared publicationsScott Jared Heidbrink
Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
6 shared publicationsUS Department of Homeland Security (DHS)
6 shared publicationsJina Lee
Sandia National Laboratories (SNL-CA), Livermore, CA (United States)
5 shared publications
Latest Publications
- Open Access
- 1 February 2025
- Published by Office of Scientific and Technical Information (OSTI)
Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models
- 1 February 2025
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessReview of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models
- Open Access
- 1 April 2024
- Published by Office of Scientific and Technical Information (OSTI)
A Framework for Evaluating the Implementation Cost of Attacks on Large Language Models
- 1 April 2024
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessA Framework for Evaluating the Implementation Cost of Attacks on Large Language Models
- Open Access
- 1 March 2024
- Published by Office of Scientific and Technical Information (OSTI)
Mini Report: LLMs for Vulnerability Repair in Code
- 1 March 2024
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessMini Report: LLMs for Vulnerability Repair in Code
- Open Access
- 1 February 2024
- Published by Office of Scientific and Technical Information (OSTI)
Mini Report: Jailbreaking Attacks and Defenses
- 1 February 2024
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessMini Report: Jailbreaking Attacks and Defenses
- Open Access
- 1 January 2024
- Published by Office of Scientific and Technical Information (OSTI)
Mini Report: LLMs on Code Translation
- 1 January 2024
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessMini Report: LLMs on Code Translation
- Open Access
- 1 December 2023
- Published by Office of Scientific and Technical Information (OSTI)
Emergent Behaviors and the Limits of Large Language Model Generalization
- 1 December 2023
- Published by Office of Scientific and Technical Information (OSTI)
Open AccessEmergent Behaviors and the Limits of Large Language Model Generalization
- Open Access
- 12 December 2020
- Published by Wiley in Cognitive Science
- Vol. 44 (12) , e12921
Do Programmers Prefer Predictable Expressions in Code?
Source code is a form of human communication, albeit one where the information shared between the programmers reading and writing the code is constrained by the requirement that the code executes correctly. Programming languages are more syntactically constrained than natural languages, but they are also very expressive, allowing a great many different ways to express even very simple computations. Still, code written by developers is highly predictable, and many programming tools have taken advantage of this phenomenon, relying on language model surprisal as a guiding mechanism. While surprisal has been validated as a measure of cognitive load in natural language, its relation to human cognitive processes in code is still poorly understood. In this paper, we explore the relationship between surprisal and programmer preference at a small granularity—do programmers prefer more predictable expressions in code? Using meaning‐preserving transformations, we produce equivalent alternatives to developer‐written code expressions and run a corpus study on Java and Python projects. In general, language models rate the code expressions developers choose to write as more predictable than these transformed alternatives. Then, we perform two human subject studies asking participants to choose between two equivalent snippets of Java code with different surprisal scores (one original and transformed). We find that programmers do prefer more predictable variants, and that stronger language models like the transformer align more often and more consistently with these preferences.
- 12 December 2020
- Published by Wiley in Cognitive Science
- Vol. 44 (12) , e12921
Open AccessDo Programmers Prefer Predictable Expressions in Code?
Source code is a form of human communication, albeit one where the information shared between the programmers reading and writing the code is constrained by the requirement that the code executes correctly. Programming languages are more syntactically constrained than natural languages, but they are also very expressive, allowing a great many different ways to express even very simple computations. Still, code written by developers is highly predictable, and many programming tools have taken advantage of this phenomenon, relying on language model surprisal as a guiding mechanism. While surprisal has been validated as a measure of cognitive load in natural language, its relation to human cognitive processes in code is still poorly understood. In this paper, we explore the relationship between surprisal and programmer preference at a small granularity—do programmers prefer more predictable expressions in code? Using meaning‐preserving transformations, we produce equivalent alternatives to developer‐written code expressions and run a corpus study on Java and Python projects. In general, language models rate the code expressions developers choose to write as more predictable than these transformed alternatives. Then, we perform two human subject studies asking participants to choose between two equivalent snippets of Java code with different surprisal scores (one original and transformed). We find that programmers do prefer more predictable variants, and that stronger language models like the transformer align more often and more consistently with these preferences.
- Open Access
Do People Prefer "Natural" code?
Natural code is known to be very repetitive (much more so than natural language corpora); furthermore, this repetitiveness persists, even after accounting for the simpler syntax of code. However, programming languages are very expressive, allowing a great many different ways (all clear and unambiguous) to express even very simple computations. So why is natural code repetitive? We hypothesize that the reasons for this lie in fact that code is bimodal: it is executed by machines, but also read by humans. This bimodality, we argue, leads developers to write code in certain preferred ways that would be familiar to code readers. To test this theory, we 1) model familiarity using a language model estimated over a large training corpus and 2) run an experiment applying several meaning preserving transformations to Java and Python expressions in a distinct test corpus to see if forms more familiar to readers (as predicted by the language models) are in fact the ones actually written. We find that these transformations generally produce program structures that are less common in practice, supporting the theory that the high repetitiveness in code is a matter of deliberate preference. Finally, 3) we use a human subject study to show alignment between language model score and human preference for the first time in code, providing support for using this measure to improve code.
Open AccessDo People Prefer "Natural" code?
Natural code is known to be very repetitive (much more so than natural language corpora); furthermore, this repetitiveness persists, even after accounting for the simpler syntax of code. However, programming languages are very expressive, allowing a great many different ways (all clear and unambiguous) to express even very simple computations. So why is natural code repetitive? We hypothesize that the reasons for this lie in fact that code is bimodal: it is executed by machines, but also read by humans. This bimodality, we argue, leads developers to write code in certain preferred ways that would be familiar to code readers. To test this theory, we 1) model familiarity using a language model estimated over a large training corpus and 2) run an experiment applying several meaning preserving transformations to Java and Python expressions in a distinct test corpus to see if forms more familiar to readers (as predicted by the language models) are in fact the ones actually written. We find that these transformations generally produce program structures that are less common in practice, supporting the theory that the high repetitiveness in code is a matter of deliberate preference. Finally, 3) we use a human subject study to show alignment between language model score and human preference for the first time in code, providing support for using this measure to improve code.
- 11 January 2019
- Published by Springer Nature in Empirical Software Engineering
- Vol. 24 (4) , 1823-1868
Studying the difference between natural and programming language corpora
- 11 January 2019
- Published by Springer Nature in Empirical Software Engineering
- Vol. 24 (4) , 1823-1868
Studying the difference between natural and programming language corpora