The Development of Small-Scale Language Models for Low-Resource Languages, with a Focus on Kazakh and Direct Preference Optimization
Top Cited Papers
Open Access
- 20 May 2025
- journal article
- research article
- Published by MDPI AG in Big Data and Cognitive Computing
- Vol. 9 (5) , 137
Abstract
Low-resource languages remain underserved by contemporary large language models (LLMs) because they lack sizable corpora, bespoke preprocessing tools, and the computing budgets assumed by mainstream alignment pipelines. Focusing on Kazakh, we present a 1.94B parameter LLaMA-based model that demonstrates how strong, culturally aligned performance can be achieved without massive infrastructure. The contribution is threefold. (i) Data and tokenization—we compile a rigorously cleaned, mixed-domain Kazakh corpus and design a tokenizer that respects the language’s agglutinative morphology, mixed-script usage, and diacritics. (ii) Training recipe—the model is built in two stages: causal language modeling from scratch followed by instruction tuning. Alignment is further refined with Direct Preference Optimization (DPO), extended by contrastive and entropy-based regularization to stabilize training under sparse, noisy preference signals. Two complementary resources support this step: ChatTune-DPO, a crowd-sourced set of human preference pairs, and Pseudo-DPO, an automatically generated alternative that repurposes instruction data to reduce annotation cost. (iii) Evaluation and impact—qualitative and task-specific assessments show that targeted monolingual training and the proposed DPO variant markedly improve factuality, coherence, and cultural fidelity over baseline instruction-only and multilingual counterparts. The model and datasets are released under open licenses, offering a reproducible blueprint for extending state-of-the-art language modeling to other under-represented languages and domains.Keywords
Funding Information
- Ministry of Science and Higher Education of the Republic of Kazakhstan (BR24993001)
- Creation of a large language model (LLM) to maintain the implementation of Kazakh language and increase the technological progress (BR24993001)
This publication has 15 references indexed in Scilit:
- ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat TemplatesProceedings of the AAAI Conference on Artificial Intelligence, 2025
- On Continually Tracing Origins of LLM-Generated Text and Its Application in Detecting Cheating in Student CourseworkBig Data and Cognitive Computing, 2025
- A Survey on the Use of Large Language Models (LLMs) in Fake NewsFuture Internet, 2024
- Prompt Engineering for Knowledge Creation: Using Chain-of-Thought to Support Students’ Improvable IdeasAI, 2024
- Optimizing Curriculum Vitae Concordance: A Comparative Examination of Classical Machine Learning Algorithms and Large Language Model ArchitecturesAI, 2024
- Analyzing the Impact of a Structured LLM Workshop in Different Education LevelsApplied Sciences, 2024
- From Large Language Models to Large Multimodal Models: A Literature ReviewApplied Sciences, 2024
- Integration AI Techniques in Low-Resource Language: The Case of Kazakh Language2024
- Axiological approach as a factor of university curriculum languageXLinguae, 2024
- RoFormer: Enhanced Transformer with Rotary Position EmbeddingarXiv, 2023