Briana B. Morrison

The use of subgoal labels in introduction to programming classrooms has been shown to improve student performance, learning, retention, and reduce students' drop out rates. However, creating and adding subgoal labels to programming assignments is often hard to articulate and very time-intensive for instructors. In Computing Education Research, Large Language Models (LLMs) have been widely used to generate human-like outputs such as worked examples and source code. In this work, we explore whether ChatGPT could be used to generate high-quality and appropriate subgoal labels in two programming curricula. Our qualitative data analysis suggests that LLMs can assist instructors in creating subgoal labels in their classrooms, opening up directions to empower students' learning experience in programming classrooms.

Training teachers to teach high-quality computer science courses is an important step towards increasing participation in computer science. To meet this need, our Research Practitioner Partnership (RPP) was formed with the primary goal of creating a comprehensive, graduate-level program to train in-service CS teachers with little to no prior CS coursework. Since its formation, the RPP has iteratively designed, implemented, and evaluated a five-course program to improve participants' knowledge of computer science content and pedagogy while allowing them to earn the state's grades 5-12 computer science endorsement. Our program has successfully scaled from a single-site pilot to a truly statewide, multi-site program, emphasizing educator-based, standards-based, and cohort-based instruction. Currently, the RPP serves 250 in-service participants. In this paper, we explain the curriculum development process and delivery model for our program. In doing so, we discuss how the program was intentionally developed from the ground up by designing course offerings that align with the CSTA student and educator standards. We make a case for peer-based Communities of Practice (CoPs) as an essential element of such programs. Finally, we discuss feedback and lessons learned during our ongoing curriculum development process with the hope that these lessons may be valuable for similar organizations looking to create a comparable, comprehensive, CS teacher training program.

Programming is a complex task that requires the development of many skills including knowledge of syntax, problem decomposition, algorithm development, and debugging. Code-writing activities are commonly used to help students develop these skills, but the difficulty of writing code from a blank page can overwhelm many novices. Parsons problems offer a simpler alternative to writing code by providing scrambled code blocks that must be placed in the correct order to solve a problem. In the 16 years since their introduction to the computing education community, an expansive body of literature has emerged that documents a range of tools, novel problem variations and makes numerous claims of benefits to learners. In this work, we track the origins of Parsons problems, outline their defining characteristics, and conduct a comprehensive review of the literature to document the evidence of benefits to learners and to identify gaps that require exploration. To facilitate future work, we design empirical studies and develop associated resources that are ready for deployment at a large scale. Collectively, this review and the provided experimental resources will serve as a focal point for researchers interested in advancing our understanding of Parsons problems and their benefits to learners.

The subgoal learning framework has improved performance for novice programmers in higher education, but it has only started to be applied and studied in K-12 (primary/secondary). Programming education in K-12 is growing, and many international initiatives are attempting to increase participation, including curricular initiatives like Computer Science Principles and non-profit organizations like Code.org. Given that subgoal learning is designed to help students with no prior knowledge, we designed and implemented subgoals in the introduction to programming unit in Code.org's Computer Science Principles course. The redesigned unit includes subgoal-oriented instruction and subgoal-themed pre-written comments that students could add to their programming activities. To evaluate efficacy, we compared behaviors and performance of students who received the redesigned subgoal unit to those receiving the original unit. We found that students who learned with subgoals performed better on problem-solving questions but not knowledge-based questions and wrote more in open-ended response questions, including a practice Performance Task for the AP exam. Moreover, at least one-third of subgoal students continued to use the subgoal comments after the subgoal-oriented instruction had been faded, suggesting that they found them useful. Survey data from the teachers suggested that students who struggled with the concepts found the subgoals most useful. Implications for future designs are discussed.

Improving upon Best Practices: FCAR 2.0The Faculty Course Assessment Report (FCAR) presents a streamlined methodology that allowsinstructors to write assessment reports in a concise, standardized format conducive for use inboth course and student (program) outcomes assessment. The FCAR is a short (typically 1-2pages) form, completed by the instructor that taught the class. The FCAR is structured as asequence of standardized reporting categories that include what course modifications were made,the outcomes assessment information obtained, reflection on the part of the instructor, andsuggestions for curricular improvement. Through this approach, the instructor is guided througha systematic review of the course, with the additional benefit of clearly and succinctlydocumenting critical portions of the "closing the loop" process. At the center of this approach isthe concept of performance vectors, a 4-tuple vector that categorizes aggregate studentperformance (in terms of the number of students who performed at excellent, adequate, minimal,or unsatisfactory levels) on a directly measured assessment artifact. For each performancecriterion to be reported on, an entry is placed into the FCAR documenting the criterion, theoutcome being supported, the assignment(s) used for acquiring the data, the assessment tool usedfor evaluating the data, and the resultant performance vector. Additionally, as this assessmentinformation is processed by the instructor who is closest to the data, any observed difficulties orextenuating circumstances affecting performance can be readily documented as part of theFCAR. For the department chair and/or assessment coordinator, FCARs provide a valuableresource, as all assessment information regarding a particular course is included in one place in acommon format providing ease of use. This information can then be extracted and summarizedin a way that allows all courses that cover a given Student Outcome to be easily evaluated.For the past several years, presentations featuring the FCAR methodology have been given at theABET Symposium. Based partly on this dissemination stream, the use of this instrument hasspread far beyond its origins. As other institutions have adopted the instrument, they have mademodifications as well, and in turn reported at a variety of venues regarding how their version ofthe FCAR has played a highly useful role in streamlining their continuous quality improvementprocesses, yielding both qualitative and quantitative information, facilitating greater consistencyin the reporting and processing of that information, and keeping faculty actively engaged in anon-going assessment process. In brief, this paper will present a survey of the literature regardingthe use of FCARs, provide an in-depth description of the current implementation of thisinstrument in at least 5 departments at 3 universities, and present recommendations for a model“FCAR 2.0” document. The paper will include example copies of FCARs from variousinstitutions, details on their use, and faculty responses on the positive and negative aspects ofimplementing this approach to assessment.

Background and Context: Subgoal labeled worked examples have been extensively researched, but the research has been reported piecemeal. This paper aggregates data from three studies, including data previously unreported, to holistically examine the effect of subgoal labeled worked examples across three student populations and across different instructional designs. Objective: By aggregating the data, we provide more statistical power for somewhat surprising yet replicable results. We discuss which results generalize across populations, focusing on a stable effect size for subgoal labels in programming instruction. Method: We use descriptive and inferential statistics to examine the data collected from different student populations and different classroom instructional designs. We concentrate on the effect size across samples of the intervention for generalization. Findings: Students using two variations of subgoal labeled instructional materials perform better than the others: the group that was given the subgoal labels with farther transfer between worked examples and practice problems and the group that constructed their own subgoal labels with nearer transfer between worked examples and practice problems.