Rickard Stureborg
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Rickard Stureborg has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Bhuwan Dhingra
Duke University
4 shared publications - Jun Yang
Duke University, Durham, NC, USA
3 shared publications - Dimitris Alikaniotis2 shared publications
- Yoshi Suhara
Harvard University
2 shared publications Aayushi Patel
Duke University
1 shared publication- Chloe Qinyu Zhu
Duke University
1 shared publication
Latest Publications
- 23 August 2024
- Published by Springer Nature
- p. 249-264
Hierarchical Multi-label Classification of Online Vaccine Concerns
- 23 August 2024
- Published by Springer Nature
- p. 249-264
Hierarchical Multi-label Classification of Online Vaccine Concerns
- Open Access
Large Language Models are Inconsistent and Biased Evaluators
The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.
Open AccessLarge Language Models are Inconsistent and Biased Evaluators
The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 76-89
Characterizing the Confidence of Large Language Model-Based Automatic Evaluation Metrics
Rickard Stureborg, Dimitris Alikaniotis, Yoshi Suhara. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 76-89
Open AccessCharacterizing the Confidence of Large Language Model-Based Automatic Evaluation Metrics
Rickard Stureborg, Dimitris Alikaniotis, Yoshi Suhara. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 2553-2575
Tailoring Vaccine Messaging with Common-Ground Opinions
Rickard Stureborg, Sanxing Chen, Roy Xie, Aayushi Patel, Christopher Li, Chloe Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra. Findings of the Association for Computational Linguistics: NAACL 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 2553-2575
Open AccessTailoring Vaccine Messaging with Common-Ground Opinions
Rickard Stureborg, Sanxing Chen, Roy Xie, Aayushi Patel, Christopher Li, Chloe Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra. Findings of the Association for Computational Linguistics: NAACL 2024. 2024.
- Open Access
- 19 April 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-17
Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations
Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how concept hierarchies can inform the design of annotation interfaces to improve labeling quality and efficiency. We study this question through annotation of vaccine misinformation, where the labeling task is difficult and highly subjective. We investigate 6 user interface designs for crowdsourcing hierarchical labels by collecting over 18,000 individual annotations. Under a fixed budget, integrating hierarchies into the design improves crowdsource workers’ F1 scores. We attribute this to (1) Grouping similar concepts, improving F1 scores by +0.16 over random groupings, (2) Strong relative performance on high-difficulty examples (relative F1 score difference of +0.40), and (3) Filtering out obvious negatives, increasing precision by +0.07. Ultimately, labeling schemes integrating the hierarchy outperform those that do not — achieving mean F1 of 0.70.
- 19 April 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-17
Open AccessInterface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations
Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how concept hierarchies can inform the design of annotation interfaces to improve labeling quality and efficiency. We study this question through annotation of vaccine misinformation, where the labeling task is difficult and highly subjective. We investigate 6 user interface designs for crowdsourcing hierarchical labels by collecting over 18,000 individual annotations. Under a fixed budget, integrating hierarchies into the design improves crowdsource workers’ F1 scores. We attribute this to (1) Grouping similar concepts, improving F1 scores by +0.16 over random groupings, (2) Strong relative performance on high-difficulty examples (relative F1 score difference of +0.40), and (3) Filtering out obvious negatives, increasing precision by +0.07. Ultimately, labeling schemes integrating the hierarchy outperform those that do not — achieving mean F1 of 0.70.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 239-244
Exploring the Effect of Frequency Resolution in FNet
Gregory Szumel, Ghazal Khalighinejad, Rickard Stureborg, Sam Wiseman. Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 239-244
Open AccessExploring the Effect of Frequency Resolution in FNet
Gregory Szumel, Ghazal Khalighinejad, Rickard Stureborg, Sam Wiseman. Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). 2023.