Rohan Alexander

Generative AI is used for knowledge work but many current AI evaluations focus on isolated tasks and constituent parts such as writing code, rather than end-to-end workflows. We test whether AI models can generate end-to-end data science projects. This requires AI models to select a dataset, develop a research question, write code to download, clean, and test data, create figures, and write a reproducible paper. To do this we collect 40 real end-to-end data science projects, then grade them according to a rubric. We use these to build an automated grading pipeline that evaluates data science projects and aligns with the grades that human graders give. For each of nine AI models we then generate five projects and use the automated grader to evaluate them. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment such as explaining measurement, writing abstracts, and citing sources. Anthropic’s Claude Opus 4.8 averaged 87 per cent and OpenAI’s GPT-5.4 averaged 85 per cent, both around the level of an A for an undergraduate. No model was consistently the best across all aspects of the rubric. These findings suggest that while AI models could soon approximate entry-level data scientists on routine tasks, they require consistent verification and evaluation.

Silicon sampling involves providing a large language model (LLM) with a persona, which is typically some demographic or attitudinal information, then asking a question, and evaluating the response. It has been found to be accurate in some situations and is increasingly used in place of, or as a complement to, evaluations of real humans because it is relatively fast and cheap. Based on data from the 2025 US Cooperative Election Study we systematically evaluate how different AI models, persona detail, news diets, and prompt styles affect synthetic responses. We find that different configurations can cause the share of respondents who support the Democrats to vary across the entire 0 to 100 per cent range. Even if we restrict the configurations to just one model, Democrat support can range from 8 to 100 per cent. The news diet changes the Democrat share by about 10 percentage points unless political identity is included in the persona in which case it has little effect. Our findings motivate the need for disclosure statements to accompany any silicon sample detailing the settings under which it was generated.

Federal representation in Canada is structurally geographical: one Member of Parliament (MP) represents one electoral district. Despite this, Canadians can legally donate to a number of political entities, regardless of proximity. Using Elections Canada's Open Data on Contributions matched with socio-demographic data from the Canadian Census, we characterize broad patterns in Canadian out-of-district donation from 2015 to 2024. We find that out-of-district donations form a persistent share of donor-sourced funding to both candidates and district associations, averaging 40% across time, election periods, and parties. While relatively constant through time, the propensity for out-of-district donation varied substantially across districts, and more than what we would expect from their spatial arrangement alone. Districts with higher population density, median household income, educational attainment, and visible minority proportions see higher rates of out-of-district giving, the same traits that predict donor prevalence generally. Still, donor-sourced local campaign funding remains largely regionalized; nearly half of donations that leave their district are received by a political entity in a neighbouring one, and few cross provincial lines. These and other findings may inform further research into the geography of local campaign finance, in Canada and other Westminster-style parliamentary democracies.

Journalists have always had to make the challenging decision about which stories to focus on and which stories to ignore. This paper investigates whether AI can assist journalists in identifying newsworthy stories from large databases. We first build a dataset of Canadian lobbying registrations, each labeled as "newsworthy" or "not newsworthy" based on whether a journalist covered it in a published article or not. We then systematically evaluate several AI models' ability to identify the one newsworthy registration within a batch of ten. The best models identify a single newsworthy registration with 22% accuracy, rising to 48% when allowed to select three, and to 52% when provided previous headline examples. Our results suggest that a filtering approach silently discards real stories, whereas allowing the model to rank registrations by newsworthiness ensures no information is lost. We conclude that while AI holds real potential to help identify leads, this is not possible without further investment in evaluation infrastructure, and even then, there is an important role for journalists to identify what AI misses, as it cannot replace their editorial judgment entirely.

Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.

We systematically evaluate the reproducibility of data analysis conducted by Large Language Models (LLMs). We evaluate two prompting strategies, six models, and four temperature settings, with ten independent executions per configuration, yielding 480 total attempts. We assess the completion, concordance, validity, and consistency of each attempt and find considerable variation in the analytical results even for consistent configurations. This suggests, as with human data analysis, the data analysis conducted by LLMs can vary, even given the same task, data, and settings. Our results mean that if an LLM is being used to conduct data analysis, then it should be run multiple times independently and the distribution of results considered.

Data science is an integrated workflow of technical, analytical, communication, and ethical skills, but current AI benchmarks focus mostly on constituent parts. We test whether AI models can generate end-to-end data science projects. To do this we create a benchmark of 40 end-to-end data science projects with associated rubric evaluations. We use these to build an automated grading pipeline that systematically evaluates the data science projects produced by generative AI models. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment. These findings suggest that while AI models could approximate entry-level data scientists on routine tasks, they require verification.

Large Language Models (LLMs) have shifted in just a few years from novelty to ubiquity, raising fundamental questions for data science education. Tasks once used to teach coding, writing, and problem-solving can now be completed by LLMs, forcing educators to reconsider both pedagogy and assessment. To understand how instructors are adapting, we conducted semi-structured interviews with 42 instructors from 33 institutions in 10 countries in June and July 2025. Our qualitative analysis reveals a pragmatic mix of optimism and concern. Many respondents view LLMs as inevitable classroom tools -- comparable to calculators or Wikipedia -- while others worry about de-skilling, misplaced confidence, and uneven integration across institutions. Around 58 per cent have already introduced demonstrations, guided activities, or make extensive use of LLMs in their courses, though most expect change to remain slow and uneven. That said, 31 per cent have not used LLMs to teach students and do not plan to. We highlight some instructional innovations, including AI-aware assessments, reflective use of LLMs as tutors, and course-specific chatbots. By sharing these perspectives, we aim to help data science educators adapt collectively to ensure curricula keep pace with technological change.

Which Canadian Members of Parliament (MPs) are on Bluesky and what types of content do they share? Taking up calls for more mere description of how emerging social media platforms are used in their initial period of operation, this research note describes how many MPs are using Bluesky and what types of content they share. Of the 123 MPs already on Bluesky, we find that they apply the same logic and understanding of platform affordances from Twitter (now X), with posts most frequently discussing policy, the Ottawa bubble and their constituency. This research note contributes to our understanding of how MPs use Bluesky to communicate with the public in a high-choice media environment.