Introducing Bode: A Fine-Tuned Large Language Model for Portuguese Prompt-Based Task
Preprint
Abstract
Large Language Models (LLMs) are increasingly bringing advances to Natural Language Processing. However, low-resource languages, those lacking extensive prominence in datasets for various NLP tasks, or where existing datasets are not as substantial, such as Portuguese, already obtain several benefits from LLMs, but not to the same extent. LLMs trained on multilingual datasets normally struggle to respond to prompts in Portuguese satisfactorily, presenting, for example, code switching in their responses. This work proposes a fine-tuned LLaMA 2-based model for Portuguese prompts named Bode in two versions: 7B and 13B. We evaluate the performance of this model in classification tasks using the zero-shot approach with in-context learning, and compare it with other LLMs. Our main contribution is to bring an LLM with satisfactory results in the Portuguese language, as well as to provide a model that is free for research or commercial purposes.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints2023
- ClueWeb22: 10 Billion Web Documents with Rich Information2022
- GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow2021
- A Survey Of zero shot detection: Methods and applicationsCognitive Robotics, 2021
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019