Rahat Rizvi Rahman

Toxic interactions in open source discussions can alienate contributors and threaten project sustainability, yet prior empirical studies of GitHub toxicity have been limited in scale, raising questions about their generalizability. Scaling up is difficult because toxicity on GitHub is often implicit and context-dependent, making both fully manual annotation and LLM-based labeling unreliable. We present a human-in-the-loop (HITL) annotation methodology that makes large-scale, domain-calibrated toxicity labeling practical. A single call to a small, local LLM produces both a toxicity prediction and a set of interpretable event category scores. A lightweight Random Forest validator then uses those scores to flag the small subset of conversations most likely to be mislabeled, directing human review only where it is needed. The validator outperforms confidence-based and multi-LLM baselines while adding low annotation cost. We apply this pipeline to over 124,000 GitHub issue and pull request conversations. Using the resulting dataset, we evaluate key findings from prior small-scale research, confirming some and qualifying others, and present new insights into the prevalence, characteristics, and dynamics of toxicity across diverse open source projects.

Emotions play a key role in collaborative software practices like pair programming. In this case study, we analyze pair programming videos to assess whether Large Language Models (LLMs) can serve as pair programming partners. We adapt the EmpatheticDialogues benchmark to create scenarios that simulate how an LLM responds in emotional moments in pair programming. We find that LLMs generally match the level of empathy humans provide in emotional moments during pair programming. However, they sometimes get lost in conversational dynamics and miss the big picture, and prioritize task progression over discussion. Our findings highlight both the potential and current limitations of LLMs in emotionally sensitive programming contexts.