Self Supervised Bert for Legal Text Classification
Authors:
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Critical BERT-based text classification tasks, such as legal text classification, require huge amounts of accurately labeled data. Legal text classification faces two trivial problems: labeling legal data is a sensitive process and can only be carried out by skilled professionals, and legal text is prone to privacy issues hence not all the data can be made available in the public domain. This means that we have limited diversity in the textual data, and to account for this data paucity, we propose a self-supervision approach to train Legal-BERT classifiers. We use the BERT text classifier’s knowledge of the class boundaries and perform gradient ascent w.r.t. class logits. Synthetic latent texts are generated through activation maximization. The main advantages over existing SOTAs are that our model: is easy to train, does not require much data but instead uses the synthesized data as fake samples; has less variance that helps to generate texts with good sample quality and diversity. We show the efficacy of the proposed method on the ECHR Violation (Multi-Label) Dataset and the Over-ruling Task Dataset.Keywords
This publication has 16 references indexed in Scilit:
- BERT-PLI: Modeling Paragraph-Level Interactions for Legal Case Retrieval2020
- LEGAL-BERT: The Muppets straight out of Law School2020
- Long and Diverse Text Generation with Planning-based Hierarchical Variational Model2019
- Neural Legal Judgment Prediction in English2019
- Multi-Task Deep Learning for Legal Document Translation, Summarization and Multi-Label Classification2018
- Empirical Study of Deep Learning for Text Classification in Legal Document Review2018
- Interpretable Textual Neuron Representations for NLP2018
- Interpretable Charge Predictions for Criminal Cases: Learning to Generate Court Views from Fact Descriptions2018
- Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space2017
- A Diversity-Promoting Objective Function for Neural Conversation Models2016