JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge
Preprint
Abstract
We introduce JBE-QA, a Japanese Bar Exam Question-Answering dataset to evaluate large language models' legal knowledge. Derived from the multiple-choice (tanto-shiki) section of the Japanese bar exam (2015-2024), JBE-QA provides the first comprehensive benchmark for Japanese legal-domain evaluation of LLMs. It covers the Civil Code, the Penal Code, and the Constitution, extending beyond the Civil Code focus of prior Japanese resources. Each question is decomposed into independent true/false judgments with structured contextual fields. The dataset contains 3,464 items with balanced labels. We evaluate 26 LLMs, including proprietary, open-weight, Japanese-specialised, and reasoning models. Our results show that proprietary models with reasoning enabled perform best, and the Constitution questions are generally easier than the Civil Code or the Penal Code questions.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Named entity recognition on Indonesian legal documents: a dataset and study using transformer-based modelsInternational Journal of Electrical and Computer Engineering (IJECE), 2024
- Japanese tort-case dataset for rationale-supported legal judgment predictionArtificial Intelligence and Law, 2024
- IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning2024
- LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain2023
- Attentive deep neural networks for legal document retrievalArtificial Intelligence and Law, 2022
- EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain2022
- ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US2022
- LexGLUE: A Benchmark Dataset for Legal Language Understanding in English2022
- MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer2021
- JEC-QA: A Legal-Domain Question Answering DatasetProceedings of the AAAI Conference on Artificial Intelligence, 2020