AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments
Open Access
- 3 March 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 136-172
Abstract
In oral arguments, judges probe attorneys with questions about the factual record, legal claims, and the strength of their arguments. To prepare for this questioning, both law schools and practicing attorneys rely on moot courts: practice simulations of appellate hearings. Leveraging a dataset of U.S. Supreme Court oral argument transcripts, we examine whether AI models can effectively simulate justice-specific questioning for moot court-style training. Evaluating oral argument simulation is challenging because there is no single correct question for any given turn. Instead, effective questioning should reflect a combination of desirable qualities, such as anticipating substantive legal issues, detecting logical weaknesses, and maintaining an appropriately adversarial tone. We introduce a two-layer evaluation framework that assesses both the realism and pedagogical usefulness of simulated questions using complementary proxy metrics. We construct and evaluate both prompt-based and agentic oral argument simulators. We find that simulated questions are often perceived as realistic by human annotators and achieve high recall of ground truth substantive legal issues. However, models still face substantial shortcomings, including low diversity in question types and sycophancy. Importantly, these shortcomings would remain undetected under naive evaluation approaches.Keywords
All Related Versions
Funding Information
- Princeton University Language+Intelligence Seed Grant (N/A)
- Schmidt Sciences Humanities and AI Virtual Institute (N/A)
This publication has 18 references indexed in Scilit:
- Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research ToolsJournal of Empirical Legal Studies, 2025
- A Reasoning-Focused Legal Retrieval Benchmark2025
- 𝛿-Stance: A Large-Scale Real World Dataset of Stances in Legal Argumentation2025
- LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rights2024
- Exploring the state of the art in legal QA systemsJournal of Big Data, 2023
- A survey on legal question–answering systemsComputer Science Review, 2023
- Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement2023
- Inferring the Winning Party in the Supreme Court from the Pattern of Questioning at Oral ArgumentThe Journal of Legal Studies, 2010
- The Influence of Oral Arguments on the U.S. Supreme CourtAmerican Political Science Review, 2006
- Oral Advocacy and the Re-emergence of a Supreme Court BarJournal of Supreme Court History, 2005