JustEva: A Toolkit to Evaluate LLM Fairness in Legal Knowledge Inference
Authors:
- Zongyue XueZXZongyue Xue
- Siyuan ZhengSZSiyuan Zheng
- Shaochun WangSWShaochun Wang
- Yiran HuYHYiran Hu
- Shenran WangSWShenran Wang
- Yuxin YaoYYYuxin Yao
- Haitao LiHLHaitao Li
- Qingyao AiQAQingyao Ai
- Yiqun LiuYLYiqun Liu
- Yun LiuYLYun Liu
- Weixing ShenWSWeixing Shen
Preprint
Abstract
The integration of Large Language Models (LLMs) into legal practice raises pressing concerns about judicial fairness, particularly due to the nature of their "black-box" processes. This study introduces JustEva, a comprehensive, open-source evaluation toolkit designed to measure LLM fairness in legal tasks. JustEva features several advantages: (1) a structured label system covering 65 extra-legal factors; (2) three core fairness metrics - inconsistency, bias, and imbalanced inaccuracy; (3) robust statistical inference methods; and (4) informative visualizations. The toolkit supports two types of experiments, enabling a complete evaluation workflow: (1) generating structured outputs from LLMs using a provided dataset, and (2) conducting statistical analysis and inference on LLMs' outputs through regression and other statistical methods. Empirical application of JustEva reveals significant fairness deficiencies in current LLMs, highlighting the lack of fair and trustworthy LLM legal tools. JustEva offers a convenient tool and methodological foundation for evaluating and improving algorithmic fairness in the legal domain.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- LLM Diagnostic Toolkit: Evaluating LLMs for Ethical Issues2024
- Intrinsic Self-correction for Enhanced Morality: An Analysis of Internal Mechanisms and the Superficial Hypothesis2024
- FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing2022
- Does Sentence Type and Length Matter? Interactions of Age, Race, Ethnicity, and Gender on Jail and Prison SentencesCriminal Justice and Behavior, 2019
- Intersectionality of Race, Ethnicity, Gender, and Age on Criminal PunishmentSociological Perspectives, 2016
- Big Data s Disparate Impact