Cognitive Network Science Reveals Bias in GPT-3, GPT-3.5 Turbo, and GPT-4 Mirroring Math Anxiety in High-School Students
Top Cited Papers
Open Access
- 27 June 2023
- journal article
- research article
- Published by MDPI AG in Big Data and Cognitive Computing
- Vol. 7 (3) , 124
Abstract
Large Language Models (LLMs) are becoming increasingly integrated into our lives. Hence, it is important to understand the biases present in their outputs in order to avoid perpetuating harmful stereotypes, which originate in our own flawed ways of thinking. This challenge requires developing new benchmarks and methods for quantifying affective and semantic bias, keeping in mind that LLMs act as psycho-social mirrors that reflect the views and tendencies that are prevalent in society. One such tendency that has harmful negative effects is the global phenomenon of anxiety toward math and STEM subjects. In this study, we introduce a novel application of network science and cognitive psychology to understand biases towards math and STEM fields in LLMs from ChatGPT, such as GPT-3, GPT-3.5, and GPT-4. Specifically, we use behavioral forma mentis networks (BFMNs) to understand how these LLMs frame math and STEM disciplines in relation to other concepts. We use data obtained by probing the three LLMs in a language generation task that has previously been applied to humans. Our findings indicate that LLMs have negative perceptions of math and STEM fields, associating math with negative concepts in 6 cases out of 10. We observe significant differences across OpenAI’s models: newer versions (i.e., GPT-4) produce 5× semantically richer, more emotionally polarized perceptions with fewer negative associations compared to older versions and high-school students. These findings suggest that advances in the architecture of LLMs may lead to increasingly less biased models that could even perhaps someday aid in reducing harmful stereotypes in society rather than perpetuating them.
Keywords
This publication has 54 references indexed in Scilit:
- CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICONComputational Intelligence, 2012
- The Development and Part Validation of a U.K. Scale for Mathematics AnxietyJournal of Psychoeducational Assessment, 2011
- Search in External and Internal SpacesPsychological Science, 2008
- The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathologyDevelopment and Psychopathology, 2005
- Math Anxiety: Personal, Educational, and Cognitive ConsequencesCurrent Directions in Psychological Science, 2002
- Measuring individual differences in implicit cognition: The implicit association test.Journal of Personality and Social Psychology, 1998
- The Moral Character of Mad Scientists: A Cultural Critique of ScienceScience, Technology, & Human Values, 1992
- The Nature, Effects, and Relief of Mathematics AnxietyJournal for Research in Mathematics Education, 1990
- A circumplex model of affect.Journal of Personality and Social Psychology, 1980
- A spreading-activation theory of semantic processing.Psychological Review, 1975