Eric Martínez

One of the central promises of legal AI is to automate drudgery—the formal, repetitive tasks of lawyers' work that consume time without calling for much discretion. Yet it remains an open question how well AI models actually perform on such tasks. This article presents the first empirical examination of AI performance on perhaps the most ubiquitous and lamented form of legal drudgery: citation formatting under the Bluebook. We make four contributions. First, we develop a new benchmark of 2,058 Bluebook queries and show that, on average, frontier language models produce a fully compliant legal citation only 42.6% of the time in a zero-shot setting. Second, we conduct an experiment with five top law reviews and show that even a "reasoning" model falls far below the average score of the human candidates in these journals' annual editor-selection competitions. Third, we show that simply providing the models with the rules offers only modest improvements, calling into question the ability of retrieval-augmented generation (RAG) to ensure rule-following alone. Finally, we develop an approach that does meaningfully improve compliance: a neuro-symbolic system that first uses a model to parse natural language into structured citation elements, and then delegates the formatting to a deterministic rule-execution engine. This approach achieves an average accuracy increase of 32.4 percentage points and total accuracy of up to 85.5% on our benchmark. These results point toward a reorientation for legal AI. The original promise of automating drudgery still remains out of reach for even frontier language models on their own—but pairing them with symbolic rule engines may offer a tractable path forward.

Whereas principles of communicative efficiency and legal doctrine dictate that laws be comprehensible to the common world, empirical evidence suggests legal documents are largely incomprehensible to lawyers and laypeople alike. Here, a corpus analysis ( n = 59) million words) first replicated and extended prior work revealing laws to contain strikingly higher rates of complex syntactic structures relative to six baseline genres of English. Next, two preregistered text generation experiments ( n = 286) tested two leading hypotheses regarding how these complex structures enter into legal documents in the first place. In line with the magic spell hypothesis , we found people tasked with writing official laws wrote in a more convoluted manner than when tasked with writing unofficial legal texts of equivalent conceptual complexity. Contrary to the copy-and-edit hypothesis, we did not find evidence that people editing a legal document wrote in a more convoluted manner than when writing the same document from scratch. From a cognitive perspective, these results suggest law to be a rare exception to the general tendency in human language toward communicative efficiency. In particular, these findings indicate law’s complexity to be derived from its performativity, whereby low-frequency structures may be inserted to signal law’s authoritative, world-state-altering nature, at the cost of increased processing demands on readers. From a law and policy perspective, these results suggest that the tension between the ubiquity and impenetrability of the law is not an inherent one, and that laws can be simplified without a loss or distortion of communicative content.

Over the last 50 years, there have been efforts on behalf of the U.S. government to simplify legal documents for society at large. However, there has been no systematic evaluation of how effective these efforts-collectively referred to as the "plain-language movement"-have been. Here we report the results of a large-scale longitudinal corpus analysis (n ≈ 225 million words), in which we compared every law passed by congress with a comparably sized sample of English texts from four different baseline genres published during approximately the same time period. We also compared the entirety of the U.S. Code (the official compilation of all federal legislation currently in force) with a large sample of recently published texts from six baseline genres of English. We found that laws remain laden with features associated with psycholinguistic complexity-including center-embedding, passive voice, low-frequency jargon, capitalization, and sentence length-relative to the baseline genres of English, and that the prevalence of most of these features has not meaningfully declined since the initial onset of the plain-language efforts. These findings suggest top-down efforts to simplify legal texts have thus far remained largely ineffectual, despite the apparent tractability of these changes, and call into question the coherence and legitimacy of legal doctrines whose validity rests on the notion of laws being easily interpretable by laypeople. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Across modern civilization, societal norms and rules are established and communicated largely in the form of written laws. Despite their prevalence and importance, legal documents have long been widely acknowledged to be difficult to understand for those who are required to comply with them (i.e., everyone). Why? Across two preregistered experiments, we evaluated five hypotheses for why lawyers write in a complex manner. Experiment 1 revealed that lawyers, like laypeople, were less able to recall and comprehend legal content drafted in a complex “legalese” register than content of equivalent meaning drafted in a simplified register. Experiment 2 revealed that lawyers rated simplified contracts as equally enforceable as legalese contracts, and rated simplified contracts as preferable to legalese contracts on several dimensions–including overall quality, appropriateness of style, and likelihood of being signed by a client. These results suggest that lawyers who write in a convoluted manner do so as a matter of convenience and tradition as opposed to an outright preference and that simplifying legal documents would be both tractable and beneficial for lawyers and nonlawyers alike.

Across modern civilization, societal norms and rules are established and communicated largely in the form of written laws. Despite their prevalence and importance, legal documents have long been widely acknowledged to be difficult to understand by those who are required to comply with them (i.e. everyone). Why? Across two pre-registered experiments, we evaluated five hypotheses for why lawyers write in a complex manner. Experiment 1 revealed that lawyers, like laypeople, were less able to recall and comprehend legal content drafted in a complex “legalese” register than content of equivalent meaning drafted in a simplified register. Experiment 2 revealed that lawyers rated simplified contracts as equally enforceable as legalese contracts, and rated simplified contracts as preferable to legalese contracts on several dimensions–including overall quality, appropriateness of style, and likelihood of being signed by a client. These results suggest that lawyers who write in a convoluted manner do so as a matter of convenience and tradition as opposed to an outright preference, and that simplifying legal documents would be both tractable and beneficial for lawyers and non-lawyers alike.

Over the last 50 years, there have been efforts on behalf of the US government to simplify public legal documents for the benefit of society at large . However, there has been no systematic evaluation of how effective these efforts--collectively referred to as the ``plain language movement''--have been. Here we report the results of a large-scale longitudinal corpus analysis (n$\approx$225 million words), in which we compare every law passed by congress between 1951 to 2009 (as well as concurrent resolutions and proclamations), with a comparably sized sample of English texts from four different genres published during the same time period. We find that laws remain laden with features associated with processing difficulty--including center-embedding, passive voice, low-frequency jargon and capitalization--relative to each of the four baseline genres of English, and that the prevalence of these features has not meaningfully declined since the onset of the plain language movement (in some cases, their prevalence has increased). These findings suggest that top-down efforts to simplify legal language have thus far remained largely ineffectual, despite the apparent tractability of these changes, raising and informing difficult questions of law and public policy.

To what extent, if any, should the law protect sentient artificial intelligence (that is, AI that can feel pleasure or pain)? Here we surveyed United States adults (n= 1,061) on their views regarding granting 1) general legal protection, 2) legal personhood, and 3) standing to bring forth a lawsuit, with respect to sentient AI and eight other groups: humans in the jurisdiction, humans outside the jurisdiction, corporations, unions, non-human animals, the environment, humans living in the near future, and humans living in the far future. Roughly one-third of participants endorsed granting personhood and standing to sentient AI (assuming its existence) in at least some cases, the lowest of any group surveyed on, and rated the desired level of protection for sentient AI as lower than all groups other than corporations. We further investigated and observed political differences in responses; liberals were more likely to endorse legal protection and personhood for sentient AI than conservatives. Taken together, these results suggest that laypeople are not by-and-large in favor of granting legal protection to AI, and that the ordinary conception of legal status, similar to codified legal doctrine, is not based on a mere capacity to feel pleasure and pain. At the same time, the observed political differences suggest that previous literature regarding political differences in empathy and moral circle expansion apply to artificially intelligent systems and extend partially, though not entirely, to legal consideration, as well.