Soyoung Oh
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Soyoung Oh has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Bing He
Georgia Institute of Technology, Atlanta, GA, USA
1 shared publication - Gaurav Verma
Georgia Institute of Technology, Atlanta, GA, USA
1 shared publication - Srijan Kumar
Georgia Institute of Technology
1 shared publication - Vera Demberg
Department of Language Science and Technology, Saarland University, Saarbrücken, Germany
1 shared publication - Yeon-Chang Lee
Ulsan National Institute of Science and Technology (UNIST), Ulsan, Republic of Korea
1 shared publication - Yibo Hu
Georgia Institute of Technology, Atlanta, GA, USA
1 shared publication
Latest Publications
- Open Access
- 23 April 2025
- Published by The Royal Society in Royal Society Open Science
- Vol. 12 (4) , 241229
Robustness of large language models in moral judgements
With the advent of large language models (LLMs), there has been a growing interest in analysing the preferences encoded in LLMs in the context of morality. Recent work has tested LLMs on various moral judgement tasks and drawn conclusions regarding the alignment between LLMs and humans. The present contribution critically assesses the validity of the method and results employed in previous work for eliciting moral judgements from LLMs. We find that previous results are confounded by biases in the presentation of the options in moral judgement tasks and that LLM responses are highly sensitive to prompt formulation variants as simple as changing ‘Case 1’ and ‘Case 2’ to ‘(A)’ and ‘(B)’. Our results hence indicate that previous conclusions on moral judgements of LLMs cannot be upheld. We make recommendations for more sound methodological setups for future studies.
- 23 April 2025
- Published by The Royal Society in Royal Society Open Science
- Vol. 12 (4) , 241229
Open AccessRobustness of large language models in moral judgements
With the advent of large language models (LLMs), there has been a growing interest in analysing the preferences encoded in LLMs in the context of morality. Recent work has tested LLMs on various moral judgement tasks and drawn conclusions regarding the alignment between LLMs and humans. The present contribution critically assesses the validity of the method and results employed in previous work for eliciting moral judgements from LLMs. We find that previous results are confounded by biases in the presentation of the options in moral judgement tasks and that LLM responses are highly sensitive to prompt formulation variants as simple as changing ‘Case 1’ and ‘Case 2’ to ‘(A)’ and ‘(B)’. Our results hence indicate that previous conclusions on moral judgements of LLMs cannot be upheld. We make recommendations for more sound methodological setups for future studies.
- Open Access
- 29 November 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 19 (1) , 1-30
A Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creators
Online misinformation poses a global risk with significant real-world consequences. To combat misinformation, current research relies on professionals like journalists and fact-checkers for annotating and debunking false information while also developing automated machine learning methods for detecting misinformation. Complementary to these approaches, recent research has increasingly concentrated on utilizing the power of ordinary social media users, a.k.a. “the crowd,” who act as eyes-on-the-ground proactively questioning and countering misinformation. Notably, recent studies show that 96% of counter-misinformation responses originate from them. Acknowledging their prominent role, we present the first systematic and comprehensive survey of research papers that actively leverage the crowds to combat misinformation. In this survey, we first identify 88 papers related to crowd-based efforts, 1 following a meticulous annotation process adhering to the PRISMA framework (preferred reporting items for systematic reviews and meta-analyses). We then present key statistics related to misinformation, counter-misinformation, and crowd input in different formats and topics. Upon holistic analysis of the papers, we introduce a novel taxonomy of the roles played by the crowds in combating misinformation: (i) crowds as annotators who actively identify misinformation; (ii) crowds as evaluators who assess counter-misinformation effectiveness; (iii) crowds as creators who create counter-misinformation. This taxonomy explores the crowd’s capabilities in misinformation detection, identifies the prerequisites for effective counter-misinformation, and analyzes crowd-generated counter-misinformation. In each assigned role, we conduct a detailed analysis to categorize the specific utilization of the crowd. Particularly, we delve into (i) distinguishing individual, collaborative, and machine-assisted labeling for annotators; (ii) analyzing the effectiveness of counter-misinformation through surveys, interviews, and in-lab experiments for evaluators; and (iii) characterizing creation patterns and creator profiles for creators. Finally, we conclude this survey by outlining potential avenues for future research in this field.
- 29 November 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Knowledge Discovery From Data
- Vol. 19 (1) , 1-30
Open AccessA Survey on the Role of Crowds in Combating Online Misinformation: Annotators, Evaluators, and Creators
Online misinformation poses a global risk with significant real-world consequences. To combat misinformation, current research relies on professionals like journalists and fact-checkers for annotating and debunking false information while also developing automated machine learning methods for detecting misinformation. Complementary to these approaches, recent research has increasingly concentrated on utilizing the power of ordinary social media users, a.k.a. “the crowd,” who act as eyes-on-the-ground proactively questioning and countering misinformation. Notably, recent studies show that 96% of counter-misinformation responses originate from them. Acknowledging their prominent role, we present the first systematic and comprehensive survey of research papers that actively leverage the crowds to combat misinformation. In this survey, we first identify 88 papers related to crowd-based efforts, 1 following a meticulous annotation process adhering to the PRISMA framework (preferred reporting items for systematic reviews and meta-analyses). We then present key statistics related to misinformation, counter-misinformation, and crowd input in different formats and topics. Upon holistic analysis of the papers, we introduce a novel taxonomy of the roles played by the crowds in combating misinformation: (i) crowds as annotators who actively identify misinformation; (ii) crowds as evaluators who assess counter-misinformation effectiveness; (iii) crowds as creators who create counter-misinformation. This taxonomy explores the crowd’s capabilities in misinformation detection, identifies the prerequisites for effective counter-misinformation, and analyzes crowd-generated counter-misinformation. In each assigned role, we conduct a detailed analysis to categorize the specific utilization of the crowd. Particularly, we delve into (i) distinguishing individual, collaborative, and machine-assisted labeling for annotators; (ii) analyzing the effectiveness of counter-misinformation through surveys, interviews, and in-lab experiments for evaluators; and (iii) characterizing creation patterns and creator profiles for creators. Finally, we conclude this survey by outlining potential avenues for future research in this field.