Amey Karkare

With the increasing adoption of Large Language Models (LLMs), there are proposals to replace human Teaching Assistants (TAs) with LLM-based AI agents for providing feedback to students. In this paper, we explore a new hybrid model where human TAs receive AI-generated feedback for CS1 programming exercises, which they can then review and modify as needed. We conducted a large-scale randomized intervention with 185 CS1 undergraduate students, comparing the efficacy of this hybrid approach against manual feedback and direct AI-generated feedback. Our initial hypothesis predicted that AI-augmented feedback would improve TA efficiency and increase the accuracy of guidance to students. However, our findings revealed mixed results. Although students perceived improvements in feedback quality, the hybrid model did not consistently translate to better student performance. We also observed complacency among some TAs who over-relied on LLM generated feedback and failed to identify and correct inaccuracies. These results suggest that augmenting human tutors with AI may not always result in improved teaching outcomes, and further research is needed to ensure it is truly effective.

Smart Contracts (SCs) handle transactions in the Ethereum blockchain worth millions of United States dollars, making them a lucrative target for attackers seeking to exploit vulnerabilities and steal funds. The Ethereum community has developed a rich set of tools to detect vulnerabilities in SCs, including reentrancy (RE) and unhandled exceptions (UX). A dataset of SCs labelled with vulnerabilities is needed to evaluate the tools' efficacy. Existing SC datasets with labelled vulnerabilities have limitations, such as covering only a limited range of vulnerability scenarios and containing incorrect labels. As a result, there is a lack of a standardized dataset to compare the performances of these tools. SCRUBD aims to fill this gap. We present a dataset of real-world SCs and synthesized SCs labelled with RE and UX. The real-world SC dataset is labelled through crowdsourcing, followed by manual inspection by an expert, and covers both RE and UX vulnerabilities. On the other hand, the synthesized dataset is carefully crafted to cover various RE scenarios only. Using SCRUBD we compared the performance of six popular vulnerability detection tools. Based on our study, we found that Slither outperforms other tools on a crowdsourced dataset in detecting RE vulnerabilities, while Sailfish outperforms other tools on a manually synthesized dataset for detecting RE. For UX vulnerabilities, Slither outperforms all other tools.

Vulnerability assessment forms the foundation for remediation activities that an organization develops as a part of risk assessment and mitigation. This paper presents a synchrophasor measurement device-specific cyber security vulnerability assessment at network level. A synchrophasor network is essential for wide-area measurement system (WAMS) that assembles time-synchronized data from multiple power network components and facilitates the processing and transferring of these data. Attackers can leverage their knowledge of firmware and protocols to disrupt or critically damage the functioning of the power system in a stealthy manner. Nevertheless, the exploitation can be curbed or controlled if the operator is aware of the vulnerabilities associated with the field devices. The authors detail their analysis on the following aspects: (a) classification of attacks in synchrophasor network, (b) Type of resources and reconnaissance sufficient to launch an attack on field devices, (c) identification, demonstration, and exploitation of synchrophasor device vulnerabilities. The work showcases exploitable vulnerabilities, to understand the breath and scope of a synchrophasor measurement device’s exposure to a possible cyber attack. The reverse-engineered attack focuses on capturing essential features of packets and consuming the bandwidth during the three-way handshake among legitimate entities. The latency in packet transmission is gradually increased, which results in retransmission; thus, the legitimate connection gets terminated. Thus, the paper provides situational awareness at the device level and credible information regarding loopholes and weak links in field devices.

In an introductory programming (CS1) context, a Refute question asks students for a counter-example which proves that a given code fragment is an incorrect solution for a given task. Such a question can be used as an assessment item to (formatively) develop or (summatively) demonstrate a student's abilities to comprehend the task and the code well enough to recognize a mismatch. These abilities assume greater significance with the emergence of generative AI technologies capable of writing code that is plausible (at least to novice programmers) but not always correct. Instructors must address three concerns while designing an effective Refute question, each influenced by their specific teaching-learning context: (1) Is the task comprehensible? (2) Is the incorrect code a plausible solution for the task? (3) Is the complexity of finding a counter-example acceptable? While the first concern can often be addressed by reusing tasks from previous code writing questions, addressing the latter concerns may require substantial instructor effort. We therefore investigate whether concerns (2) and (3) can be addressed by buggy student solutions for the corresponding code writing question from a previous course offering. For 6 code writing questions (from a Fall 2015 C programming course), our automated evaluation system logged 13,847 snapshots of executable student code, of which 10,574 were buggy (i.e., they failed at least one instructor-supplied test case). Code selected randomly from this pool rarely addresses these concerns, and manual selection is infeasible. Our paper makes three contributions. First, we propose an automated mechanism to filter this pool to a more manageable number of snapshots from which appropriate code can be selected manually. Second, we evaluate our semi-automated mechanism with respect to concerns (2) and (3) by surveying a diverse set of 56 experienced participants (instructors, tutors, and teaching assistants). Third, we use this mechanism to seed a public repository of Refute questions and provide a template to create additional questions using a public resource (CodeCheck).

Automated pedagogical error repair (APER) is the task of suggesting fixes to buggy programs written by beginner or novice programmers. APER tools have been shown to greatly improve the learning experience for students for whom error messages offered by compilers or runtime environments are either unhelpful and often misleading. Consequently, several APER tools have been proposed in literature using a variety of powerful machine learning techniques including sequence-to-sequence modelling (TRACER), reinforcement learning (RLAssist), graph attention (DrRepair) and decision trees (MACER). Despite offering high repair rates, these tools are often bulky, requiring several days of training and extensive GPU resources. This paper describes CAPER, a novel APER tool for the C programming language that offers 4-5% higher repair accuracy than existing APER tools on multiple benchmark error repair datasets. CAPER has the added advantage of being significantly lighter and faster than most existing tools, being able to train on a CPU and yet offering training speedups of 2 × over TRACER, 600 × over RLAssist and 700 × over DrRepair. The paper also describes PyPER, an extension of CAPER to the Python programming language. Code for CAPER is available at https://github.com/purushottamkar/caper/

This is a short report on the Tutorials of the 13th Innovations in Software Engineering (ISEC 2020) conference held on 27th February 2020 in Jabalpur, India. ISEC tutorials are 3-hour sessions providing a hands-on introduction to an emerging research area or a useful research tool on any topic relevant to software engineering. The tutorials are mainly geared towards students and young researchers and are designed to equip them with tools that can help their own research and foster new research directions. The tutorials at ISEC offer a glimpse of the state-of-the-art research in various disciplines of software engineering. This year, ISEC tutorial track attracted a total of three submissions. Two submissions were selected based on the content and relevance to the theme of the conference.