Karen D. Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Karen D. Wang has a total of 17 co-authors. Here are the top co-authors by number of shared publications:
Carl Wieman
Stanford University 94305 Stanford California USA
5 shared publications- Carl Wieman
Physics Department and Graduate School of Education, Stanford University, Stanford, CA 94305, USA
3 shared publications - Shima Salehi
Graduate School of Education, Stanford University, Stanford, CA, United States
3 shared publications - Eric Burkholder
Auburn University
2 shared publications Jake McCool
Cornell University Ithaca New York USA
2 shared publicationsNick Haber
Stanford University 94305 Stanford California USA
2 shared publications
Latest Publications
- Open Access
Scaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education
Developing problem-solving competency is central to Science, Technology, Engineering, and Mathematics (STEM) education, yet translating this priority into effective approaches to problem-solving instruction and assessment remain a significant challenge. The recent proliferation of generative artificial intelligence (genAI) tools like ChatGPT in higher education introduces new considerations about how these tools can help or hinder students' development of STEM problem-solving competency. Our research examines these considerations by studying how and why college students use genAI tools in their STEM coursework, focusing on their problem-solving support. We surveyed 40 STEM college students from diverse U.S. institutions and 28 STEM faculty to understand instructor perspectives on effective genAI tool use and guidance in STEM courses. Our findings reveal high adoption rates and diverse applications of genAI tools among STEM students. The most common use cases include finding explanations, exploring related topics, summarizing readings, and helping with problem-set questions. The primary motivation for using genAI tools was to save time. Moreover, over half of student participants reported simply inputting problems for AI to generate solutions, potentially bypassing their own problem-solving processes. These findings indicate that despite high adoption rates, students' current approaches to utilizing genAI tools often fall short in enhancing their own STEM problem-solving competencies. The study also explored students' and STEM instructors' perceptions of the benefits and risks associated with using genAI tools in STEM education. Our findings provide insights into how to guide students on appropriate genAI use in STEM courses and how to design genAI-based tools to foster students' problem-solving competency.
Open AccessScaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education
Developing problem-solving competency is central to Science, Technology, Engineering, and Mathematics (STEM) education, yet translating this priority into effective approaches to problem-solving instruction and assessment remain a significant challenge. The recent proliferation of generative artificial intelligence (genAI) tools like ChatGPT in higher education introduces new considerations about how these tools can help or hinder students' development of STEM problem-solving competency. Our research examines these considerations by studying how and why college students use genAI tools in their STEM coursework, focusing on their problem-solving support. We surveyed 40 STEM college students from diverse U.S. institutions and 28 STEM faculty to understand instructor perspectives on effective genAI tool use and guidance in STEM courses. Our findings reveal high adoption rates and diverse applications of genAI tools among STEM students. The most common use cases include finding explanations, exploring related topics, summarizing readings, and helping with problem-set questions. The primary motivation for using genAI tools was to save time. Moreover, over half of student participants reported simply inputting problems for AI to generate solutions, potentially bypassing their own problem-solving processes. These findings indicate that despite high adoption rates, students' current approaches to utilizing genAI tools often fall short in enhancing their own STEM problem-solving competencies. The study also explored students' and STEM instructors' perceptions of the benefits and risks associated with using genAI tools in STEM education. Our findings provide insights into how to guide students on appropriate genAI use in STEM courses and how to design genAI-based tools to foster students' problem-solving competency.
- Open Access
- 18 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 416-425
Can Crowdsourcing Platforms Be Useful for Educational Research?
- 18 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 416-425
Open AccessCan Crowdsourcing Platforms Be Useful for Educational Research?
- Open Access
- 18 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 498-506
Discovering Players’ Problem-Solving Behavioral Characteristics in a Puzzle Game through Sequence Mining
Digital games offer promising platforms for assessing student higher-order competencies such as problem-solving. However, processing and analyzing the large volume of interaction log data generated in these platforms to uncover meaningful behavioral patterns remain a complex research challenge. In this study, we employ sequence mining and clustering techniques to examine students’ log data in an interactive puzzle game that requires player to change rules to win the game. Our goal is to identify behavioral characteristics associated with the problem-solving practices adopted by individual students. The findings indicate that the most effective problem solvers made fewer rule changes and took longer time to make those changes across both an introductory and a more advanced level of the game. Conversely, rapid rule change actions were linked to ineffective problem-solving. This research underscores the potential of sequence mining and cluster analysis as generalizable methods for understanding student higher-order competencies through log data in digital gaming and learning environments. It also suggests future directions on how to provide just-in-time, in-game feedback to enhance student problem-solving competences.
- 18 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 498-506
Open AccessDiscovering Players’ Problem-Solving Behavioral Characteristics in a Puzzle Game through Sequence Mining
Digital games offer promising platforms for assessing student higher-order competencies such as problem-solving. However, processing and analyzing the large volume of interaction log data generated in these platforms to uncover meaningful behavioral patterns remain a complex research challenge. In this study, we employ sequence mining and clustering techniques to examine students’ log data in an interactive puzzle game that requires player to change rules to win the game. Our goal is to identify behavioral characteristics associated with the problem-solving practices adopted by individual students. The findings indicate that the most effective problem solvers made fewer rule changes and took longer time to make those changes across both an introductory and a more advanced level of the game. Conversely, rapid rule change actions were linked to ineffective problem-solving. This research underscores the potential of sequence mining and cluster analysis as generalizable methods for understanding student higher-order competencies through log data in digital gaming and learning environments. It also suggests future directions on how to provide just-in-time, in-game feedback to enhance student problem-solving competences.
- 21 February 2024
- Published by Wiley in Journal of Research in Special Educational Needs
- Vol. 24 (3) , 505-518
Exploring the learning experiences of neurodivergent college students in STEM courses
Neurodivergent students exhibit an inclination towards Science, Technology, Engineering and Mathematics (STEM) fields, yet their learning experiences in STEM courses remain underexamined. Utilizing an online survey of neurodivergent ( n = 60) and neurotypical ( n = 83) US college students, this study identified various factors influencing their self‐perceived learning experiences, including interest in the course content, instruction quality and performance outcomes. Compared to their neurotypical peers, neurodivergent students attributed negative experiences in STEM courses less frequently to performance‐related factors and more often to a mismatch between their interests and the course content. Both groups also articulated a variety of strengths and challenges encountered in their STEM studies. Neurodivergent students were more likely to report having interest and passion for STEM and less likely to report having peer support and effective study skills and habits as their primary strength for studying STEM. Conversely, while neurotypical students cited difficult content as their central challenge, neurodivergent students more commonly faced challenges with focus and attention. Despite the study's limited sample size, it revealed emerging patterns that emphasize the importance of developing inclusive teaching methods and specific support mechanisms to cater to the unique strengths and challenges of neurodivergent students in higher education.
- 21 February 2024
- Published by Wiley in Journal of Research in Special Educational Needs
- Vol. 24 (3) , 505-518
Exploring the learning experiences of neurodivergent college students in STEM courses
Neurodivergent students exhibit an inclination towards Science, Technology, Engineering and Mathematics (STEM) fields, yet their learning experiences in STEM courses remain underexamined. Utilizing an online survey of neurodivergent ( n = 60) and neurotypical ( n = 83) US college students, this study identified various factors influencing their self‐perceived learning experiences, including interest in the course content, instruction quality and performance outcomes. Compared to their neurotypical peers, neurodivergent students attributed negative experiences in STEM courses less frequently to performance‐related factors and more often to a mismatch between their interests and the course content. Both groups also articulated a variety of strengths and challenges encountered in their STEM studies. Neurodivergent students were more likely to report having interest and passion for STEM and less likely to report having peer support and effective study skills and habits as their primary strength for studying STEM. Conversely, while neurotypical students cited difficult content as their central challenge, neurodivergent students more commonly faced challenges with focus and attention. Despite the study's limited sample size, it revealed emerging patterns that emphasize the importance of developing inclusive teaching methods and specific support mechanisms to cater to the unique strengths and challenges of neurodivergent students in higher education.
- Top Cited PapersOpen Access
- 18 January 2024
- Published by Frontiers Media SA in Frontiers in Education
- Vol. 8, 1330486
Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving
The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: (1) failure to construct accurate models of the physical world, (2) failure to make reasonable assumptions about missing data, and (3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.
- 18 January 2024
- Published by Frontiers Media SA in Frontiers in Education
- Vol. 8, 1330486
Top Cited PapersOpen AccessExamining the potential and pitfalls of ChatGPT in science and engineering problem-solving
The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: (1) failure to construct accurate models of the physical world, (2) failure to make reasonable assumptions about missing data, and (3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.
- Open Access
- 12 December 2023
- Published by Center for Open Science in EdArXiv
Exploring the Learning Experiences of Neurodivergent College Students in STEM Courses
Neurodivergent students exhibit an inclination toward Science, Technology, Engineering and Mathematics (STEM) fields, yet their learning experiences in STEM courses remain underexamined. Utilizing an online survey of neurodivergent (n = 60) and neurotypical (n = 83) US college students, this study identified various factors influencing their self-perceived learning experiences, including interest in the course content, instruction quality, and performance outcomes. Compared to their neurotypical peers, neurodivergent students attributed negative experiences in STEM courses less frequently to performance-related factors and more often to a mismatch between their interests and the course content. Both groups also articulated a variety of strengths and challenges encountered in their STEM studies. Neurodivergent students were more likely to report having interest and passion for STEM and less likely to report having peer support and effective study skills and habits as their primary strength for studying STEM. Conversely, while neurotypical students cited difficult content as their central challenge, neurodivergent students more commonly faced challenges with focus and attention. Despite the study's limited sample size, it revealed emerging patterns that emphasize the importance of developing inclusive teaching methods and specific support mechanisms to cater to the unique strengths and challenges of neurodivergent students in higher education.
- 12 December 2023
- Published by Center for Open Science in EdArXiv
Open AccessExploring the Learning Experiences of Neurodivergent College Students in STEM Courses
Neurodivergent students exhibit an inclination toward Science, Technology, Engineering and Mathematics (STEM) fields, yet their learning experiences in STEM courses remain underexamined. Utilizing an online survey of neurodivergent (n = 60) and neurotypical (n = 83) US college students, this study identified various factors influencing their self-perceived learning experiences, including interest in the course content, instruction quality, and performance outcomes. Compared to their neurotypical peers, neurodivergent students attributed negative experiences in STEM courses less frequently to performance-related factors and more often to a mismatch between their interests and the course content. Both groups also articulated a variety of strengths and challenges encountered in their STEM studies. Neurodivergent students were more likely to report having interest and passion for STEM and less likely to report having peer support and effective study skills and habits as their primary strength for studying STEM. Conversely, while neurotypical students cited difficult content as their central challenge, neurodivergent students more commonly faced challenges with focus and attention. Despite the study's limited sample size, it revealed emerging patterns that emphasize the importance of developing inclusive teaching methods and specific support mechanisms to cater to the unique strengths and challenges of neurodivergent students in higher education.
- Open Access
Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving
The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: 1) failure to construct accurate models of the physical world, 2) failure to make reasonable assumptions about missing data, and 3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.
Open AccessExamining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving
The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: 1) failure to construct accurate models of the physical world, 2) failure to make reasonable assumptions about missing data, and 3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.
- 14 June 2023
- Published by Springer Nature
- p. 31-52
Applying Log Data Analytics to Measure Problem Solving in Simulation-Based Learning Environments
- 14 June 2023
- Published by Springer Nature
- p. 31-52
Applying Log Data Analytics to Measure Problem Solving in Simulation-Based Learning Environments
- Open Access
- 28 November 2022
- Published by Wiley in British Journal of Educational Technology
- Vol. 54 (1) , 192-221
A systematic review of empirical studies using log data from open‐ended learning environments to measure science and engineering practices
Technology‐based, open‐ended learning environments (OELEs) can capture detailed information of students' interactions as they work through a task or solve a problem embedded in the environment. This information, in the form of log data, has the potential to provide important insights about the practices adopted by students for scientific inquiry and problem solving. How to parse and analyse the log data to reveal evidence of multifaceted constructs like inquiry and problem solving holds the key to making interactive learning environments useful for assessing students' higher‐order competencies. In this paper, we present a systematic review of studies that used log data generated in OELEs to describe, model and assess scientific inquiry and problem solving. We identify and analyse 70 conference proceedings and journal papers published between 2012 and 2021. Our results reveal large variations in OELE and task characteristics, approaches used to extract features from log data and interpretation models used to link features to target constructs. While the educational data mining and learning analytics communities have made progress in leveraging log data to model inquiry and problem solving, multiple barriers still exist to hamper the production of representative, reproducible and generalizable results. Based on the trends identified, we lay out a set of recommendations pertaining to key aspects of the workflow that we believe will help the field develop more systematic approaches to designing and using OELEs for studying how students engage in inquiry and problem‐solving practices. Practitioner notes: What is already known about this topic Research has shown that technology‐based, open‐ended learning environments (OELEs) that collect users' interaction data are potentially useful tools for engaging students in practice‐based STEM learning. More work is needed to identify generalizable principles of how to design OELE tasks to support student learning and how to analyse the log data to assess student performance. What this paper adds We identified multiple barriers to the production of sufficiently generalizable and robust results to inform practice, with respect to: (1) the design characteristics of the OELE‐based tasks, (2) the target competencies measured, (3) the approaches and techniques used to extract features from log files and (4) the models used to link features to the competencies. Based on this analysis, we can provide a series of specific recommendations to inform future research and facilitate the generalizability and interpretability of results: Making the data available in open‐access repositories, similar to the PISA tasks, for easy access and sharing. Defining target practices more precisely to better align task design with target practices and to facilitate between‐study comparisons. More systematic evaluation of OELE and task designs to improve the psychometric properties of OELE‐based measurement tasks and analysis processes. Focusing more on internal and external validation of both feature generation processes and statistical models, for example with data from different samples or by systematically varying the analysis methods. Implications for practice and/or policy Using the framework of evidence‐centered assessment design, we have identified relevant criteria for organizing and evaluating the diverse body of empirical studies on the topic and that policy makers and practitioners can use for their own further examinations. This paper identifies promising research and development areas on the measurement and assessment of higher‐order constructs with process data from OELE‐based tasks that government agencies and foundations can support. Researchers, technologists and assessment designers might find useful the insights and recommendations for how OELEs can enhance science assessment through thoughtful integration of learning theories, task design and data mining techniques.
- 28 November 2022
- Published by Wiley in British Journal of Educational Technology
- Vol. 54 (1) , 192-221
Open AccessA systematic review of empirical studies using log data from open‐ended learning environments to measure science and engineering practices
Technology‐based, open‐ended learning environments (OELEs) can capture detailed information of students' interactions as they work through a task or solve a problem embedded in the environment. This information, in the form of log data, has the potential to provide important insights about the practices adopted by students for scientific inquiry and problem solving. How to parse and analyse the log data to reveal evidence of multifaceted constructs like inquiry and problem solving holds the key to making interactive learning environments useful for assessing students' higher‐order competencies. In this paper, we present a systematic review of studies that used log data generated in OELEs to describe, model and assess scientific inquiry and problem solving. We identify and analyse 70 conference proceedings and journal papers published between 2012 and 2021. Our results reveal large variations in OELE and task characteristics, approaches used to extract features from log data and interpretation models used to link features to target constructs. While the educational data mining and learning analytics communities have made progress in leveraging log data to model inquiry and problem solving, multiple barriers still exist to hamper the production of representative, reproducible and generalizable results. Based on the trends identified, we lay out a set of recommendations pertaining to key aspects of the workflow that we believe will help the field develop more systematic approaches to designing and using OELEs for studying how students engage in inquiry and problem‐solving practices. Practitioner notes: What is already known about this topic Research has shown that technology‐based, open‐ended learning environments (OELEs) that collect users' interaction data are potentially useful tools for engaging students in practice‐based STEM learning. More work is needed to identify generalizable principles of how to design OELE tasks to support student learning and how to analyse the log data to assess student performance. What this paper adds We identified multiple barriers to the production of sufficiently generalizable and robust results to inform practice, with respect to: (1) the design characteristics of the OELE‐based tasks, (2) the target competencies measured, (3) the approaches and techniques used to extract features from log files and (4) the models used to link features to the competencies. Based on this analysis, we can provide a series of specific recommendations to inform future research and facilitate the generalizability and interpretability of results: Making the data available in open‐access repositories, similar to the PISA tasks, for easy access and sharing. Defining target practices more precisely to better align task design with target practices and to facilitate between‐study comparisons. More systematic evaluation of OELE and task designs to improve the psychometric properties of OELE‐based measurement tasks and analysis processes. Focusing more on internal and external validation of both feature generation processes and statistical models, for example with data from different samples or by systematically varying the analysis methods. Implications for practice and/or policy Using the framework of evidence‐centered assessment design, we have identified relevant criteria for organizing and evaluating the diverse body of empirical studies on the topic and that policy makers and practitioners can use for their own further examinations. This paper identifies promising research and development areas on the measurement and assessment of higher‐order constructs with process data from OELE‐based tasks that government agencies and foundations can support. Researchers, technologists and assessment designers might find useful the insights and recommendations for how OELEs can enhance science assessment through thoughtful integration of learning theories, task design and data mining techniques.