Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set
Open Access
- 25 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Abstract
Building LLMs for languages other than English is in great demand due to the unavailability and performance of multilingual LLMs, such as understanding the local context. The problem is critical for low-resource languages due to the need for instruction sets. In a multilingual country like India, there is a need for LLMs supporting Indic languages to provide generative AI and LLM-based technologies and services to its citizens. This paper presents our approach of i) generating a large Odia instruction set, including domain knowledge data suitable for LLM fine-tuning, and ii) building a Llama2-finetuned model tailored for enhanced performance in the Odia domain. The proposed work will help researchers build an instruction set and LLM, particularly for Indic languages. We will release the model and instruction set for the public for research and noncommercial purposes.Keywords
This publication has 12 references indexed in Scilit:
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes2023
- Can Large Language Models Be an Alternative to Human Evaluations?2023
- The Shape of Learning Curves: A ReviewIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory ManagementIEEE Transactions on Parallel and Distributed Systems, 2022
- Incremental PID Controller-Based Learning Rate Scheduler for Stochastic Gradient DescentIEEE Transactions on Neural Networks and Learning Systems, 2022
- Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic LanguagesTransactions of the Association for Computational Linguistics, 2022
- Reveal training performance mystery between TensorFlow and PyTorch in the single GPU environmentScience China Information Sciences, 2021
- Metrics for Automatic Evaluation of Text from NLP Models for Text to Scene GenerationEuropean Journal of Electrical Engineering and Computer Science, 2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Explaining Predictions of Non-Linear Classifiers in NLP2016