Daniela S. M. Pereira
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Daniela S. M. Pereira has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
Brian S. Lunn
1 shared publicationFilipe Falcão
1 shared publication- Francisco Ribeiro-Mourão
Paediatrics Department Unidade Local de Saúde do Alto Minho, Santa Luzia Hospital Viana do Castelo Portugal
1 shared publication José Miguel Pêgo
1 shared publication- José Miguel Pêgo
Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho
1 shared publication - João Carlos Ribeiro
Faculdade de Medicina, University of Coimbra
1 shared publication
Latest Publications
- 16 July 2024
- Published by Taylor & Francis in Medical Teacher
- Vol. 47 (4) , 677-683
ChatGPT as an item calibration tool: Psychometric insights in a high-stakes examination
ChatGPT has attracted a lot of interest worldwide for its versatility in a range of natural language tasks, including in the education and evaluation industry. It can automate time- and labor-intensive tasks with clear economic and efficiency gains. This study evaluated the potential of ChatGPT to automate psychometric analysis of test questions from the 2020 Portuguese National Residency Selection Exam (PNA). ChatGPT was queried 100 times with the 150 MCQ from the exam. Using ChatGPT's responses, difficulty indices were calculated for each question based on the proportion of correct answers. The predicted difficulty levels were compared to the actual difficulty levels of the 2020 exam MCQ's using methods from classical test theory. ChatGPT's predicted item difficulty indices positively correlated with the actual item difficulties (r (148) = -0.372, p < .001), suggesting a general consistency between the real and the predicted values. There was also a moderate significant negative correlation between the difficulty index predicted by ChatGPT and the number of challenges (r (148) = -0.302, p < .001), highlighting ChatGPT's potential for identifying less problematic questions. These findings unveiled ChatGPT's potential as a tool for assessment development, proving its capability to predict the psychometric characteristics of high-stakes test items in automated item calibration without pre-testing in real-life scenarios.
- 16 July 2024
- Published by Taylor & Francis in Medical Teacher
- Vol. 47 (4) , 677-683
ChatGPT as an item calibration tool: Psychometric insights in a high-stakes examination
ChatGPT has attracted a lot of interest worldwide for its versatility in a range of natural language tasks, including in the education and evaluation industry. It can automate time- and labor-intensive tasks with clear economic and efficiency gains. This study evaluated the potential of ChatGPT to automate psychometric analysis of test questions from the 2020 Portuguese National Residency Selection Exam (PNA). ChatGPT was queried 100 times with the 150 MCQ from the exam. Using ChatGPT's responses, difficulty indices were calculated for each question based on the proportion of correct answers. The predicted difficulty levels were compared to the actual difficulty levels of the 2020 exam MCQ's using methods from classical test theory. ChatGPT's predicted item difficulty indices positively correlated with the actual item difficulties (r (148) = -0.372, p < .001), suggesting a general consistency between the real and the predicted values. There was also a moderate significant negative correlation between the difficulty index predicted by ChatGPT and the number of challenges (r (148) = -0.302, p < .001), highlighting ChatGPT's potential for identifying less problematic questions. These findings unveiled ChatGPT's potential as a tool for assessment development, proving its capability to predict the psychometric characteristics of high-stakes test items in automated item calibration without pre-testing in real-life scenarios.
- Top Cited PapersOpen Access
- 7 September 2023
- Published by Elsevier in International Journal of Educational Research
- Vol. 122
Here's to the future: Conversational agents in higher education- a scoping review
- 7 September 2023
- Published by Elsevier in International Journal of Educational Research
- Vol. 122
Top Cited PapersOpen AccessHere's to the future: Conversational agents in higher education- a scoping review