JWJuliette Woodrow

Juliette Woodrow

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Juliette Woodrow has a total of 30 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Top Cited Papers
    Open Access

    The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement But May Increase Adopters' Exam Performances

    by Allen Nie, Yash Chandak, Miroslav Suzara, Ali Malik, Juliette Woodrow, Matt Peng, Mehran Sahami, Emma Brunskill, Chris Piech


  • Open Access

    Infinite Story

    by Chris Piech, Mehran Sahami, Yasmine Alonso, Katie Liu, Javokhir Arifov, Anjali Sreenivas, Dan Webber, Tina Zheng, Ngoc Nguyen, Iddah Mlauzi, et al.

    In Infinite Story, students build a choose-your-own-adventure game that integrates generative AI to create a dynamic, interactive "infinite story" experience. The game is powered by nested dictionary (JSON) objects that store pre-existing scenes. Each scene is a nested dictionary, containing user choices, descriptions, and more. Students are challenged to navigate and manipulate these deeply nested data structures, which helps them appreciate the utility and complexity of dictionary objects. When a user ventures into an undefined scene, the program makes an API call to ChatGPT to generate the next scene, allowing the adventure to continue seamlessly. To the best of our knowledge it is one of the first assignments in intro CS that uses ChatGPT. What makes this assignment truly nifty is how it teaches students to leverage generative AI in a creative, meaningful way. By blending generative storytelling with technical skills, students get to see the power of AI in extending their projects beyond predefined boundaries, creating an open-ended, exciting experience. Many students expanded on this assignment for their final projects, creating sophisticated programs like AI-driven Chess and Go games. Using the techniques from this assignment, they leveraged the course's OpenAI integration to build functional AI agents that enhanced gameplay. These projects showcased the flexibility of the assignment, inspiring students to think critically about the creative and practical applications of AI in real-world contexts.


  • Open Access

    TeachNow: Enabling Teachers to Provide Spontaneous, Realtime 1:1 Help in Massive Online Courses

    by Ali Malik, Juliette Woodrow, Chao Wang, Chris Piech

    One-on-one help from a teacher is highly impactful for students, yet extremely challenging to support in massive online courses (MOOCs). In this work, we present TeachNow: a novel system that lets volunteer teachers from anywhere in the world instantly provide 1:1 help sessions to students in MOOCs, without any scheduling or coordination overhead. TeachNow works by quickly finding an online student to help and putting them in a collaborative working session with the teacher. The spontaneous, on-demand nature of TeachNow gives teachers the flexibility to help whenever their schedule allows. We share our experiences deploying TeachNow as an experimental feature in a six week online CS1 course with 9,000 students and 600 volunteer teachers. Even as an optional activity, TeachNow was used by teachers to provide over 12,300 minutes of 1:1 help to 375 unique students. Through a carefully designed randomised control trial, we show that TeachNow sessions increased student course retention rate by almost 15%. Moreover, the flexibility of our system captured valuable volunteer time that would otherwise go to waste. Lastly, TeachNow was rated by teachers as one of the most enjoyable and impactful aspects of their involvement in the course. We believe TeachNow is an important step towards providing more human-centered support in massive online courses.


  • Open Access

    The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances

    by Allen Nie, Yash Chandak, Miroslav Suzara, Ali Malik, Juliette Woodrow, Matt Peng, Mehran Sahami, Emma Brunskill, Chris Piech

    Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Copilot. This type of interface is readily available to students and teachers around the world, yet relatively little research has been done to assess the impact of such generic tools on student learning. Coding education is an interesting test case, both because LLMs have strong performance on coding tasks, and because LLM-powered support tools are rapidly becoming part of the workflow of professional software engineers. To help understand the impact of generic LLM use on coding education, we conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class in which we provided some students with access to a chat interface with GPT-4. We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation. We observe similar decreases in other forms of course engagement. However, this decrease is modulated by the student's country of origin. Offering access to LLMs to students from low human development index countries increased their exam participation rate on average. Our results suggest there may be promising benefits to using LLMs in an introductory coding class, but also potential harms for engagement, which makes their longer term impact on student success unclear. Our work highlights the need for additional investigations to help understand the potential impact of future adoption and integration of LLMs into classrooms.


  • Open Access

    A Fast and Accurate Machine Learning Autograder for the Breakout Assignment

    by Evan Zheran Liu, David Yuan, Ahmed Ahmed, Elyse Cornwall, Juliette Woodrow, Kaylee Burns, Allen Nie, Emma Brunskill, Chris Piech, Chelsea Finn

    In this paper, we detail the successful deployment of a machine learning autograder that significantly decreases the grading labor required in the Breakout computer science assignment. This assignment - which tasks students with programming a game consisting of a controllable paddle and a ball that bounces off the paddle to break bricks - is popular for engaging students with introductory computer science concepts, but creates a large grading burden. Due to the game's interactive nature, grading defies traditional unit tests and instead typically requires 8+ minutes of manually playing each student's game to search for bugs. This amounts to 45+ hours of grading in a standard course offering and prevents further widespread adoption of the assignment. Our autograder alleviates this burden by playing each student's game with a reinforcement learning agent and providing videos of discovered bugs to instructors. In an A/B test with manual grading, we find that our human-in-the-loop AI autograder reduces grading time by 44%, while slightly improving grading accuracy by 6%, ultimately saving roughly 30 hours over our deployment in two offerings of the assignment. Our results further suggest the practicality of grading other interactive assignments (e.g., other games or building websites) via similar machine learning techniques. Live demo at https://ezliu.github.io/breakoutgrader.


  • Open Access

    AI Teaches the Art of Elegant Coding: Timely, Fair, and Helpful Style Feedback in a Global Course

    by Juliette Woodrow, Ali Malik, Chris Piech

    Teaching students how to write code that is elegant, reusable, and comprehensible is a fundamental part of CS1 education. However, providing this "style feedback" in a timely manner has proven difficult to scale. In this paper, we present our experience deploying a novel, real-time style feedback tool in Code in Place, a large-scale online CS1 course. Our tool is based on the latest breakthroughs in large-language models (LLMs) and was carefully designed to be safe and helpful for students. We used our Real-Time Style Feedback tool (RTSF) in a class with over 8,000 diverse students from across the globe and ran a randomized control trial to understand its benefits. We show that students who received style feedback in real-time were five times more likely to view and engage with their feedback compared to students who received delayed feedback. Moreover, those who viewed feedback were more likely to make significant style-related edits to their code, with over 79% of these edits directly incorporating their feedback. We also discuss the practicality and dangers of LLM-based tools for feedback, investigating the quality of the feedback generated, LLM limitations, and techniques for consistency, standardization, and safeguarding against demographic bias, all of which are crucial for a tool utilized by students.


  • Open Access

    Learners Teaching Novices: An Uplifting Alternative Assessment

    by Ali Malik, Juliette Woodrow, Chris Piech

    We propose and carry-out a novel method of formative assessment called Assessment via Teaching (AVT), in which learners demonstrate their understanding of CS1 topics by tutoring more novice students. AVT has powerful benefits over traditional forms of assessment: it is centered around service to others and is highly rewarding for the learners who teach. Moreover, teaching greatly improves the learners' own understanding of the material and has a huge positive impact on novices, who receive free 1:1 tutoring. Lastly, this form of assessment is naturally difficult to cheat---a critical property for assessments in the era of large-language models. We use AVT in a randomised control trial with learners in a CS1 course at an R1 university. The learners provide tutoring sessions to more novice students taking a lagged online version of the same course. We show that learners who do an AVT session before the course exam performed 20 to 30 percentage points better than the class average on several questions. Moreover, compared to students who did a practice exam, the AVT learners enjoyed their experience more and were twice as likely to study for their teaching session. We believe AVT is a scalable and uplifting method for formative assessment that could one day replace traditional exams.