Juliette Woodrow
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Juliette Woodrow has a total of 30 co-authors. Here are the top co-authors by number of shared publications:
- Chris Piech
Stanford University, Stanford, CA, USA
8 shared publications - Ali Malik
Stanford University, Stanford, CA, USA
4 shared publications - Allen Nie
Stanford University, Stanford, CA, USA
3 shared publications - Emma Brunskill
Computer Science, Stanford University, Stanford, California, USA
3 shared publications - Mehran Sahami
Stanford University, Stanford, CA, USA
2 shared publications - Ahmed Ahmed
Stanford University, Stanford, CA, USA
1 shared publication
Latest Publications
- Open Access
- 17 February 2026
- Published by Association for Computing Machinery (ACM)
- p. 610-616
Aligning Small Language Models for Programming Feedback: Towards Scalable Coding Support in a Massive Global Course
- 17 February 2026
- Published by Association for Computing Machinery (ACM)
- p. 610-616
Open AccessAligning Small Language Models for Programming Feedback: Towards Scalable Coding Support in a Massive Global Course
- Top Cited PapersOpen Access
- 17 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 376-380
The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement But May Increase Adopters' Exam Performances
- 17 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 376-380
Top Cited PapersOpen AccessThe GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement But May Increase Adopters' Exam Performances
- Open Access
- 3 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 750-760
Soft Grades: A Calibrated and Accurate Method for Course-Grade Estimation that Expresses Uncertainty
- 3 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 750-760
Open AccessSoft Grades: A Calibrated and Accurate Method for Course-Grade Estimation that Expresses Uncertainty
- Open Access
- 18 February 2025
- Published by Association for Computing Machinery (ACM)
- p. 1750
Infinite Story
In Infinite Story, students build a choose-your-own-adventure game that integrates generative AI to create a dynamic, interactive "infinite story" experience. The game is powered by nested dictionary (JSON) objects that store pre-existing scenes. Each scene is a nested dictionary, containing user choices, descriptions, and more. Students are challenged to navigate and manipulate these deeply nested data structures, which helps them appreciate the utility and complexity of dictionary objects. When a user ventures into an undefined scene, the program makes an API call to ChatGPT to generate the next scene, allowing the adventure to continue seamlessly. To the best of our knowledge it is one of the first assignments in intro CS that uses ChatGPT. What makes this assignment truly nifty is how it teaches students to leverage generative AI in a creative, meaningful way. By blending generative storytelling with technical skills, students get to see the power of AI in extending their projects beyond predefined boundaries, creating an open-ended, exciting experience. Many students expanded on this assignment for their final projects, creating sophisticated programs like AI-driven Chess and Go games. Using the techniques from this assignment, they leveraged the course's OpenAI integration to build functional AI agents that enhanced gameplay. These projects showcased the flexibility of the assignment, inspiring students to think critically about the creative and practical applications of AI in real-world contexts.
- 18 February 2025
- Published by Association for Computing Machinery (ACM)
- p. 1750
Open AccessInfinite Story
In Infinite Story, students build a choose-your-own-adventure game that integrates generative AI to create a dynamic, interactive "infinite story" experience. The game is powered by nested dictionary (JSON) objects that store pre-existing scenes. Each scene is a nested dictionary, containing user choices, descriptions, and more. Students are challenged to navigate and manipulate these deeply nested data structures, which helps them appreciate the utility and complexity of dictionary objects. When a user ventures into an undefined scene, the program makes an API call to ChatGPT to generate the next scene, allowing the adventure to continue seamlessly. To the best of our knowledge it is one of the first assignments in intro CS that uses ChatGPT. What makes this assignment truly nifty is how it teaches students to leverage generative AI in a creative, meaningful way. By blending generative storytelling with technical skills, students get to see the power of AI in extending their projects beyond predefined boundaries, creating an open-ended, exciting experience. Many students expanded on this assignment for their final projects, creating sophisticated programs like AI-driven Chess and Go games. Using the techniques from this assignment, they leveraged the course's OpenAI integration to build functional AI agents that enhanced gameplay. These projects showcased the flexibility of the assignment, inspiring students to think critically about the creative and practical applications of AI in real-world contexts.
- Open Access
- 3 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 708-714
TeachNow: Enabling Teachers to Provide Spontaneous, Realtime 1:1 Help in Massive Online Courses
One-on-one help from a teacher is highly impactful for students, yet extremely challenging to support in massive online courses (MOOCs). In this work, we present TeachNow: a novel system that lets volunteer teachers from anywhere in the world instantly provide 1:1 help sessions to students in MOOCs, without any scheduling or coordination overhead. TeachNow works by quickly finding an online student to help and putting them in a collaborative working session with the teacher. The spontaneous, on-demand nature of TeachNow gives teachers the flexibility to help whenever their schedule allows. We share our experiences deploying TeachNow as an experimental feature in a six week online CS1 course with 9,000 students and 600 volunteer teachers. Even as an optional activity, TeachNow was used by teachers to provide over 12,300 minutes of 1:1 help to 375 unique students. Through a carefully designed randomised control trial, we show that TeachNow sessions increased student course retention rate by almost 15%. Moreover, the flexibility of our system captured valuable volunteer time that would otherwise go to waste. Lastly, TeachNow was rated by teachers as one of the most enjoyable and impactful aspects of their involvement in the course. We believe TeachNow is an important step towards providing more human-centered support in massive online courses.
- 3 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 708-714
Open AccessTeachNow: Enabling Teachers to Provide Spontaneous, Realtime 1:1 Help in Massive Online Courses
One-on-one help from a teacher is highly impactful for students, yet extremely challenging to support in massive online courses (MOOCs). In this work, we present TeachNow: a novel system that lets volunteer teachers from anywhere in the world instantly provide 1:1 help sessions to students in MOOCs, without any scheduling or coordination overhead. TeachNow works by quickly finding an online student to help and putting them in a collaborative working session with the teacher. The spontaneous, on-demand nature of TeachNow gives teachers the flexibility to help whenever their schedule allows. We share our experiences deploying TeachNow as an experimental feature in a six week online CS1 course with 9,000 students and 600 volunteer teachers. Even as an optional activity, TeachNow was used by teachers to provide over 12,300 minutes of 1:1 help to 375 unique students. Through a carefully designed randomised control trial, we show that TeachNow sessions increased student course retention rate by almost 15%. Moreover, the flexibility of our system captured valuable volunteer time that would otherwise go to waste. Lastly, TeachNow was rated by teachers as one of the most enjoyable and impactful aspects of their involvement in the course. We believe TeachNow is an important step towards providing more human-centered support in massive online courses.
- Open Access
- 25 April 2024
- Published by Center for Open Science in OSF Preprints
The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances
Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Copilot. This type of interface is readily available to students and teachers around the world, yet relatively little research has been done to assess the impact of such generic tools on student learning. Coding education is an interesting test case, both because LLMs have strong performance on coding tasks, and because LLM-powered support tools are rapidly becoming part of the workflow of professional software engineers. To help understand the impact of generic LLM use on coding education, we conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class in which we provided some students with access to a chat interface with GPT-4. We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation. We observe similar decreases in other forms of course engagement. However, this decrease is modulated by the student's country of origin. Offering access to LLMs to students from low human development index countries increased their exam participation rate on average. Our results suggest there may be promising benefits to using LLMs in an introductory coding class, but also potential harms for engagement, which makes their longer term impact on student success unclear. Our work highlights the need for additional investigations to help understand the potential impact of future adoption and integration of LLMs into classrooms.
- 25 April 2024
- Published by Center for Open Science in OSF Preprints
Open AccessThe GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances
Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Copilot. This type of interface is readily available to students and teachers around the world, yet relatively little research has been done to assess the impact of such generic tools on student learning. Coding education is an interesting test case, both because LLMs have strong performance on coding tasks, and because LLM-powered support tools are rapidly becoming part of the workflow of professional software engineers. To help understand the impact of generic LLM use on coding education, we conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class in which we provided some students with access to a chat interface with GPT-4. We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation. We observe similar decreases in other forms of course engagement. However, this decrease is modulated by the student's country of origin. Offering access to LLMs to students from low human development index countries increased their exam participation rate on average. Our results suggest there may be promising benefits to using LLMs in an introductory coding class, but also potential harms for engagement, which makes their longer term impact on student success unclear. Our work highlights the need for additional investigations to help understand the potential impact of future adoption and integration of LLMs into classrooms.
- Open Access
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 736-742
A Fast and Accurate Machine Learning Autograder for the Breakout Assignment
In this paper, we detail the successful deployment of a machine learning autograder that significantly decreases the grading labor required in the Breakout computer science assignment. This assignment - which tasks students with programming a game consisting of a controllable paddle and a ball that bounces off the paddle to break bricks - is popular for engaging students with introductory computer science concepts, but creates a large grading burden. Due to the game's interactive nature, grading defies traditional unit tests and instead typically requires 8+ minutes of manually playing each student's game to search for bugs. This amounts to 45+ hours of grading in a standard course offering and prevents further widespread adoption of the assignment. Our autograder alleviates this burden by playing each student's game with a reinforcement learning agent and providing videos of discovered bugs to instructors. In an A/B test with manual grading, we find that our human-in-the-loop AI autograder reduces grading time by 44%, while slightly improving grading accuracy by 6%, ultimately saving roughly 30 hours over our deployment in two offerings of the assignment. Our results further suggest the practicality of grading other interactive assignments (e.g., other games or building websites) via similar machine learning techniques. Live demo at https://ezliu.github.io/breakoutgrader.
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 736-742
Open AccessA Fast and Accurate Machine Learning Autograder for the Breakout Assignment
In this paper, we detail the successful deployment of a machine learning autograder that significantly decreases the grading labor required in the Breakout computer science assignment. This assignment - which tasks students with programming a game consisting of a controllable paddle and a ball that bounces off the paddle to break bricks - is popular for engaging students with introductory computer science concepts, but creates a large grading burden. Due to the game's interactive nature, grading defies traditional unit tests and instead typically requires 8+ minutes of manually playing each student's game to search for bugs. This amounts to 45+ hours of grading in a standard course offering and prevents further widespread adoption of the assignment. Our autograder alleviates this burden by playing each student's game with a reinforcement learning agent and providing videos of discovered bugs to instructors. In an A/B test with manual grading, we find that our human-in-the-loop AI autograder reduces grading time by 44%, while slightly improving grading accuracy by 6%, ultimately saving roughly 30 hours over our deployment in two offerings of the assignment. Our results further suggest the practicality of grading other interactive assignments (e.g., other games or building websites) via similar machine learning techniques. Live demo at https://ezliu.github.io/breakoutgrader.
- Open Access
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 1442-1448
AI Teaches the Art of Elegant Coding: Timely, Fair, and Helpful Style Feedback in a Global Course
Teaching students how to write code that is elegant, reusable, and comprehensible is a fundamental part of CS1 education. However, providing this "style feedback" in a timely manner has proven difficult to scale. In this paper, we present our experience deploying a novel, real-time style feedback tool in Code in Place, a large-scale online CS1 course. Our tool is based on the latest breakthroughs in large-language models (LLMs) and was carefully designed to be safe and helpful for students. We used our Real-Time Style Feedback tool (RTSF) in a class with over 8,000 diverse students from across the globe and ran a randomized control trial to understand its benefits. We show that students who received style feedback in real-time were five times more likely to view and engage with their feedback compared to students who received delayed feedback. Moreover, those who viewed feedback were more likely to make significant style-related edits to their code, with over 79% of these edits directly incorporating their feedback. We also discuss the practicality and dangers of LLM-based tools for feedback, investigating the quality of the feedback generated, LLM limitations, and techniques for consistency, standardization, and safeguarding against demographic bias, all of which are crucial for a tool utilized by students.
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 1442-1448
Open AccessAI Teaches the Art of Elegant Coding: Timely, Fair, and Helpful Style Feedback in a Global Course
Teaching students how to write code that is elegant, reusable, and comprehensible is a fundamental part of CS1 education. However, providing this "style feedback" in a timely manner has proven difficult to scale. In this paper, we present our experience deploying a novel, real-time style feedback tool in Code in Place, a large-scale online CS1 course. Our tool is based on the latest breakthroughs in large-language models (LLMs) and was carefully designed to be safe and helpful for students. We used our Real-Time Style Feedback tool (RTSF) in a class with over 8,000 diverse students from across the globe and ran a randomized control trial to understand its benefits. We show that students who received style feedback in real-time were five times more likely to view and engage with their feedback compared to students who received delayed feedback. Moreover, those who viewed feedback were more likely to make significant style-related edits to their code, with over 79% of these edits directly incorporating their feedback. We also discuss the practicality and dangers of LLM-based tools for feedback, investigating the quality of the feedback generated, LLM limitations, and techniques for consistency, standardization, and safeguarding against demographic bias, all of which are crucial for a tool utilized by students.
- Open Access
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 785-791
Learners Teaching Novices: An Uplifting Alternative Assessment
We propose and carry-out a novel method of formative assessment called Assessment via Teaching (AVT), in which learners demonstrate their understanding of CS1 topics by tutoring more novice students. AVT has powerful benefits over traditional forms of assessment: it is centered around service to others and is highly rewarding for the learners who teach. Moreover, teaching greatly improves the learners' own understanding of the material and has a huge positive impact on novices, who receive free 1:1 tutoring. Lastly, this form of assessment is naturally difficult to cheat---a critical property for assessments in the era of large-language models. We use AVT in a randomised control trial with learners in a CS1 course at an R1 university. The learners provide tutoring sessions to more novice students taking a lagged online version of the same course. We show that learners who do an AVT session before the course exam performed 20 to 30 percentage points better than the class average on several questions. Moreover, compared to students who did a practice exam, the AVT learners enjoyed their experience more and were twice as likely to study for their teaching session. We believe AVT is a scalable and uplifting method for formative assessment that could one day replace traditional exams.
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 785-791
Open AccessLearners Teaching Novices: An Uplifting Alternative Assessment
We propose and carry-out a novel method of formative assessment called Assessment via Teaching (AVT), in which learners demonstrate their understanding of CS1 topics by tutoring more novice students. AVT has powerful benefits over traditional forms of assessment: it is centered around service to others and is highly rewarding for the learners who teach. Moreover, teaching greatly improves the learners' own understanding of the material and has a huge positive impact on novices, who receive free 1:1 tutoring. Lastly, this form of assessment is naturally difficult to cheat---a critical property for assessments in the era of large-language models. We use AVT in a randomised control trial with learners in a CS1 course at an R1 university. The learners provide tutoring sessions to more novice students taking a lagged online version of the same course. We show that learners who do an AVT session before the course exam performed 20 to 30 percentage points better than the class average on several questions. Moreover, compared to students who did a practice exam, the AVT learners enjoyed their experience more and were twice as likely to study for their teaching session. We believe AVT is a scalable and uplifting method for formative assessment that could one day replace traditional exams.