Veronika Hein

Background: Although generative‑AI systems are increasingly used to solve non‑routine problems, effective prompting strategies remain largely underexplored.Aims: The present study investigates how university students prompt ChatGPT to solve complex ill-defined problems, specifically examining which prompts are associated with higher or lower problem-solving performance.Sample: Seventy-seven university students (53 women; Mage = 22.4 years) participated in the study.Methods: To identify various prompt types employed by students, the study utilized qualitative analysis of interactions with ChatGPT 3.5 during the resolution of the creative problem-solving task. Participants’ performance was measured by the quality, elaboration, and originality of their ideas. Subsequently, two-step clustering was employed to identify groups of low- and high-performing students. Finally, process-mining techniques (heuristics miner) were used to analyze the interactions of low- and high-performing students.Results: The findings suggest that including clear evaluation criteria when prompting ChatGPT to generate ideas (rs = .38), providing ChatGPT with an elaborated context for idea generation (rs = .47), and offering specific feedback (rs = .45), enhances the quality, elaboration, and originality of the solutions. Successful problem-solving involves iterative human-AI regulation, with high performers using an overall larger number of prompts (d = 0.82). High performers interacted with ChatGPT through dialogue, where they monitored and regulated the generation of ideas, while low performers used ChatGPT as an information resource.Conclusions: These results emphasize the importance of active and iterative engagement for creative problem-solving and suggest that educational practices should foster metacognitive monitoring and regulation to maximize the benefits of human-AI collaboration.

Recent studies have conceptualized ChatGPT as an epistemic authority; however, no research has yet examined how epistemic beliefs and metacognitive accuracy affect students' actual use of ChatGPT‐generated content, which often contains factual inaccuracies. Therefore, the present experimental study aimed to examine how university students integrate correct and incorrect information from expert‐written and ChatGPT‐generated articles when writing independently ( N = 49) or with ChatGPT assistance ( N = 49). Students working with ChatGPT‐4o integrated more correct information from both expert‐written ( d = 0.64) and ChatGPT‐generated articles ( d = 0.95), but ChatGPT‐assisted writing did not affect the amount of incorrect information sourced from the ChatGPT‐generated article. Regardless of the condition, hierarchical regressions revealed that lower metacognitive bias was moderately associated with increased inclusion of correct information from the expert‐written article ( R ² = 12%). Conversely, a higher metacognitive bias ( R ² = 10%) and epistemic beliefs ( R ² = 12%) were moderately related to the inclusion of incorrect information from ChatGPT‐generated articles. These findings suggest that while ChatGPT assistance enhances the integration of correct human‐ and AI‐generated content, metacognitive skills remain essential to mitigate the risks of incorporating incorrect AI‐generated information. Practitioner notes: What is already known about this topic Generative AI tools, such as ChatGPT, are increasingly regarded as epistemic authorities due to their authoritative tone and human‐like interaction. ChatGPT has demonstrated utility in providing correct information and improving productivity in educational and professional contexts, but it is also prone to inaccuracies, hallucinations and misleading content. Students' epistemic beliefs and metacognitive skills predict their ability to critically evaluate and integrate conflicting information from multiple resources, particularly when searching for information on the Internet. What this paper adds This study experimentally examines how students integrate correct and incorrect information from expert‐written and ChatGPT‐generated articles when writing independently or with ChatGPT's assistance. The findings show that ChatGPT assistance improves the inclusion of correct information but does not significantly reduce or increase the inclusion of incorrect ChatGPT‐generated content. Metacognitive accuracy and epistemic beliefs are key factors in mitigating the inclusion of incorrect information, regardless of whether students work independently or with ChatGPT. Implications for practice and/or policy Generative AI tools can outperform human experts in specific scenarios, requiring little to no evaluation. However, in situations where these tools generate misleading or incorrect content, the application of metacognitive skills and epistemic beliefs becomes essential to discern reliable information and avoid the integration of errors. Educational interventions should include activities requiring justification of knowledge, evaluation of resources and reflection upon human‐generated and AI‐generated texts to enhance students' ability to discern accurate from inaccurate information. Interventions focused on metacognitive accuracy and epistemic awareness can empower individuals to critically evaluate and differentiate between reliable and erroneous information, enhancing their recognition of misinformation.

Recent studies have identified ChatGPT as an epistemic authority; however, no research has yet examined how epistemic beliefs and metacognitive accuracy impact actual work with ChatGPT-generated content, which is often prone to factual inaccuracies. Therefore, the present experimental study aimed to examine how university students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently (N = 49) or with ChatGPT assistance (N = 49). Students working with ChatGPT-4o integrated more correct information from both expert-written (d = 0.64) and ChatGPT-generated articles (d = 0.95), but ChatGPT-assisted writing did not affect the amount of incorrect information sourced from the ChatGPT-generated article. Regardless of the condition, hierarchical regressions revealed that lower metacognitive bias was moderately associated with increased inclusion of correct information from the expert-written article (R2 = 12%). Conversely, a higher metacognitive bias (R2 = 10%) and epistemic beliefs (R2 = 12%) were moderately related to the inclusion of incorrect information from ChatGPT-generated articles. These findings suggest that while ChatGPT assistance enhances the integration of correct human- and AI-generated content, metacognitive skills remain essential to mitigate the risks of incorporating incorrect AI-generated information.

University students often employ generative artificial intelligence tools such as ChatGPT in resolution of ill-defined problem-solving tasks. However, the experimental evidence about effects of ChatGPT on complex problem-solving performance is still missing. In this preregistered experiment, the impact of ChatGPT on performance in a complex creative problem-solving task was investigated in 77 university students solving a task with ChatGPT in comparison to 68 students solving a task without it. ChatGPT use significantly improved self-efficacy for task resolution (d = 0.65) and enhanced the quality (d = 0.69), elaboration (d = 0.61), and originality (d = 0.55) of solutions. Moreover, participants with ChatGPT assistance perceived task as easier (d = 0.56) and requiring less mental effort (d = 0.58). However, use of ChatGPT did not make task resolution more interesting (d = 0.08), and the impact of ChatGPT on metacognitive monitoring accuracy was unclear. Although there were no significant differences in absolute accuracy between students solving the task with and without the assistance of ChatGPT, the absence of correlation between self-evaluation judgments and performance suggests that participants struggled to calibrate their self-evaluations when using ChatGPT. Notably, the perceived usefulness of ChatGPT appeared to inform self-evaluation judgments, resulting in higher inaccuracy. The implications for hybrid human-AI regulation (HHAIR) theory are discussed. To regulate effectively, students using AI tools should focus on valid metacognitive cues and ignore cues that could lead to inaccurate outcomes.