Comparing Expert and ChatGPT-authored Guidance Prompts
Open Access
- 9 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 388-392
Abstract
Students bring a multitude of ideas and experiences to the classroom while they are reasoning about scientific phenomena. They often need timely guidance to refine build upon their initial ideas. In this study we explore the development of guidance prompts to provide students with personalized, real-time feedback in the context of a pedagogically grounded chatbot. In the current version of the tool, guidance prompts are authored by learning scientists who are experts in the content of the items and in Knowledge Integration pedagogy. When students engage with the chatbot, an idea detection model is used to determine the ideas that are present in a student explanation and then the expert-authored guidance prompts are assigned based on rules about which ideas are or are not present in the student explanation. While this approach allows for close attention to and control of the pedagogical intent of each prompt, it is time consuming and not easily generalizable. Further this rule-based approach limits the ways in which students can interact with the chatbot. The work in progress study presented in this paper explores the potential of using generative AI to create similarly pedagogically grounded guidance prompts as a first step towards increasing the generalizability and scalability of this approach. Specifically, we ask: using criteria from the Knowledge Integration Pedagogical Framework, how do ChatGPT 3.5-authored guidance prompts compare to human expert-authored guidance prompts? We find that while prompt engineering can enhance the alignment of ChatGPT-authored guidance prompts with pedagogical criteria, the human expert-authored guidance prompts more consistently meet the pedagogical criteria.Keywords
Funding Information
- National Science Foundation (2101669)
This publication has 11 references indexed in Scilit:
- Explaining Thermodynamics: Impact of an Adaptive Dialog Based on a Natural Language Processing Idea Detection Model2023
- Adaptive Dialog to Support Student Understanding of Climate Change Mechanism and Who is Most Impacted2023
- OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research2023
- Computer-based guidance to support students’ revision of their science explanationsComputers & Education, 2022
- A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University ClassroomsDigital, 2021
- An empirical investigation of neural methods for content scoring of science explanations2020
- Automated, adaptive guidance for K-12 educationEducational Research Review, 2015
- The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and TeachingInternational Journal of Artificial Intelligence in Education, 2014
- Pedagogical Practices to Support Classroom Cultures of Scientific InquiryCognition and Instruction, 2011
- The Knowledge Integration Perspective on Learning and Instruction2005