Evaluating Domain-adapted Language Models for Governmental Text Classification Tasks in Portuguese
Open Access
- 14 October 2024
- conference paper
- Published by Sociedade Brasileira de Computação
- p. 247-259
Abstract
Domain-adaptive pre-training (DAPT) is a technique in natural language processing (NLP) that tailors pre-trained language models to specific domains, enhancing their performance in real-world applications. In this paper, we evaluate the effectiveness of DAPT in governmental text classification tasks, exploring how different factors, such as target domain dataset, pre-trained model language composition, and dataset size, impact model performance. We systematically vary these factors, creating distinct domain-adapted models derived from BERTimbau and LaBSE. Our experimental results reveal that selecting appropriate target domain datasets and pre-training strategies can notably enhance the performance of language models in governmental tasks.Keywords
This publication has 17 references indexed in Scilit:
- PLUS: A Semi-automated Pipeline for Fraud Detection in Public BidsDigital Government: Research and Practice, 2024
- LegalBert-pt: A Pretrained Language Model for the Brazilian Portuguese Legal Domain2023
- Evaluating Contextualized Embeddings for Topic Modeling in Public Bidding Domain2023
- Domain Adaptation: Challenges, Methods, Datasets, and ApplicationsIEEE Access, 2023
- Detecting Inconsistencies in Public Bids: An Automated and Data-based Approach2022
- PetroBERT: A Domain Adaptation Language Model for Oil and Gas Applications in Portuguese2022
- Evaluating Topic Models in Portuguese Political Comments About Bills from Brazil’s Chamber of Deputies2021
- BERTimbau: Pretrained BERT Models for Brazilian Portuguese2020
- Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks2020
- LeNER-Br: A Dataset for Named Entity Recognition in Brazilian Legal Text2018