Evaluating LLM-Generated Contextualized Algorithm Design Problems
Open Access
- 10 August 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 407-421
Abstract
Background: Context personalization, the practice of adapting learning materials to students’ personal interests, has been shown to increase student learning and engagement. Within computer science education, research has found that LLMs can generate high-quality contextualized introductory programming exercises. Objective: In this paper, we evaluate the capability of LLMs to generate technically correct and thematically integrated contextualized algorithm design problems. Methods: In a series of three iterative studies, we use LLMs to generate contextualized algorithm design problems from a given base problem and theme, evaluating over 500 generated problems for technical and thematic alignment. Results: We find that LLM-generated algorithm design problems exhibit significantly more issues than prior work has found for introductory programming problems. We identify issues specific to the algorithm design context and then mitigate these issues with prompt engineering techniques and model choice. With these adjustments, we produce LLM-generated contextualized algorithm design problems that are technically strong, deeply themed, and largely realistic, though realism drops with more culturally and locally specific themes. Implications: We demonstrate a viable workflow for generating contextualized algorithm design problems using LLMs, including prompt design, model selection, and identification of specific issues to review for.Keywords
Funding Information
- National Science Foundation (CCF-2313998)
- National Science Foundation (CCF-2434362)
This publication has 58 references indexed in Scilit:
- Experiences and Proposals of Use of Generative AI in Advanced Software Courses2024
- Students’ interests for personalized learning: an analysis guideEuropean Journal of Psychology of Education, 2022
- The role of situational interest in personalized learning.Journal of Educational Psychology, 2018
- Personalization of Instruction: Design Dimensions and Implications for CognitionThe Journal of Experimental Education, 2017
- Eliciting Mathematics Interest: New Directions for Context Personalization and Example ChoiceThe Journal of Experimental Education, 2016
- Teaching and learning recursive programming: a review of the research literatureComputer Science Education, 2015
- Motivating Students by “Personalizing” Learning around Individual Interests: A Consideration of Theory, Design, and Implementation Issues2014
- Using adaptive learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes.Journal of Educational Psychology, 2013
- Spheres of influence: what shapes young people’s aspirations at age 12/13 and what are the implications for education policy?Journal of Education Policy, 2013
- Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice.Journal of Educational Psychology, 1996