Harshita Diddee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Harshita Diddee has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Agrima Seth
School of Information University of Michigan
2 shared publications - Daphne Ippolito
Carnegie Mellon University
2 shared publications - Kalika Bali
Microsoft Research India
2 shared publications - Rishav Hada
Microsoft Research India
2 shared publications - Aashiq Muhamed
Carnegie Mellon University Carnegie Mellon University Carnegie Mellon University Lingjing Kong Carnegie Mellon University Carnegie Mellon University Carnegie Mellon University Virginia Smith Carnegie Mellon University Carnegie Mellon University & MBZUAI
1 shared publication - Abhinav Sukumar Rao
Language Technologies Institute Carnegie Mellon University
1 shared publication
Latest Publications
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 1943-1957
Chasing Random: Instruction Selection Strategies Fail to Generalize
Harshita Diddee, Daphne Ippolito. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 1943-1957
Open AccessChasing Random: Instruction Selection Strategies Fail to Generalize
Harshita Diddee, Daphne Ippolito. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.
- Open Access
Chasing Random: Instruction Selection Strategies Fail to Generalize
Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. This has accelerated the development of methods that filter a large, noisy instruction-tuning datasets down to high-quality subset which works just as well. However, typically, the performance of these methods is not demonstrated across a uniform experimental setup and thus their generalization capabilities are not well established. In this work, we analyze popular selection strategies across different source datasets, selection budgets and evaluation benchmarks: Our results indicate that selection strategies generalize poorly, often failing to consistently outperform even random baselines. We also analyze the cost-performance trade-offs of using data selection. Our findings reveal that data selection can often exceed the cost of fine-tuning on the full dataset, yielding only marginal and sometimes no gains compared to tuning on the full dataset or a random subset.
Open AccessChasing Random: Instruction Selection Strategies Fail to Generalize
Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. This has accelerated the development of methods that filter a large, noisy instruction-tuning datasets down to high-quality subset which works just as well. However, typically, the performance of these methods is not demonstrated across a uniform experimental setup and thus their generalization capabilities are not well established. In this work, we analyze popular selection strategies across different source datasets, selection budgets and evaluation benchmarks: Our results indicate that selection strategies generalize poorly, often failing to consistently outperform even random baselines. We also analyze the cost-performance trade-offs of using data selection. Our findings reveal that data selection can often exceed the cost of fine-tuning on the full dataset, yielding only marginal and sometimes no gains compared to tuning on the full dataset or a random subset.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 37-46
Less is Fed More: Sparsity Reduces Feature Distortion in Federated Learning
Abhinav Sukumar Rao, Aashiq Muhamed, Harshita Diddee. Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 37-46
Open AccessLess is Fed More: Sparsity Reduces Feature Distortion in Federated Learning
Abhinav Sukumar Rao, Aashiq Muhamed, Harshita Diddee. Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U). 2024.
- Open Access
''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text
Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and is seen in online and offline discourses. With LLMs increasingly gaining human-like fluency in text generation, gaining a nuanced understanding of the biases these systems can generate is imperative. Prior work often treats gender bias as a binary classification task. However, acknowledging that bias must be perceived at a relative scale; we investigate the generation and consequent receptivity of manual annotators to bias of varying degrees. Specifically, we create the first dataset of GPT-generated English text with normative ratings of gender bias. Ratings were obtained using Best--Worst Scaling -- an efficient comparative annotation framework. Next, we systematically analyze the variation of themes of gender biases in the observed ranking and show that identity-attack is most closely related to gender bias. Finally, we show the performance of existing automated models trained on related concepts on our dataset.
Open Access''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text
Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the most pervasive biases in our society and is seen in online and offline discourses. With LLMs increasingly gaining human-like fluency in text generation, gaining a nuanced understanding of the biases these systems can generate is imperative. Prior work often treats gender bias as a binary classification task. However, acknowledging that bias must be perceived at a relative scale; we investigate the generation and consequent receptivity of manual annotators to bias of varying degrees. Specifically, we create the first dataset of GPT-generated English text with normative ratings of gender bias. Ratings were obtained using Best--Worst Scaling -- an efficient comparative annotation framework. Next, we systematically analyze the variation of themes of gender biases in the observed ranking and show that identity-attack is most closely related to gender bias. Finally, we show the performance of existing automated models trained on related concepts on our dataset.
- Top Cited PapersOpen Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 1862-1876
“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text
Rishav Hada, Agrima Seth, Harshita Diddee, Kalika Bali. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 1862-1876
Top Cited PapersOpen Access“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text
Rishav Hada, Agrima Seth, Harshita Diddee, Kalika Bali. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.