Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
Authors:
Preprint
Abstract
The advancement of natural language processing for morphologically rich and moderately-resourced languages like Modern Greek has been hindered by architectural stagnation, data scarcity, and limited context processing capabilities, particularly in specialized domains such as law. In this work, we propose the Greek Embedding Models (GEMs), a new family of transformer-based language models, specifically developed to address these limitations through architectural diversity and enhanced data curation. The proposed family of models are trained on several large-scale, meticulously curated corpora, encompassing both comprehensive general-domain datasets and specialized legal collections, addressing the persistent data scarcity that has impeded Greek language modeling advancement. The proposed quality-based corpus curation methodology incorporates extensive preprocessing pipelines, sophisticated deduplication strategies and targeted repetition of high-quality legal sub-corpora to enhance domain adaptation. The GEMs family comprises both established architectures (RoBERTa and Longformer) and advanced models not previously applied to Greek (ELECTRA, ConvBERT, and ModernBERT), providing comprehensive coverage of modern transformer designs. Additionally, we introduce the first bilingual Greek-English embedding models tailored for cross-lingual legal applications. Comprehensive evaluation across three core natural language understanding benchmarks demonstrates that the proposed GEM-RoBERTa and GEM-ConvBERT achieve statistically significant performance improvements over established state-of-the-art models, with accuracy gains of up to 3.6\% while conducted statistical analysis using Friedman Aligned-Ranks and Finner post-hoc tests confirms the superiority of our approach across multiple evaluation metrics.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference2025
- Adaptive augmentation framework for domain independent few shot learningKnowledge-Based Systems, 2024
- Mutual information-based neighbor selection method for causal effect estimationNeural Computing and Applications, 2024
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- From Pre-Training to Meta-Learning: A Journey in Low-Resource-Language Representation LearningIEEE Access, 2023
- DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains2023
- Multi-granular Legal Topic Classification on Greek Legislation2021
- How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models2021
- A Primer in BERTology: What We Know About How BERT WorksTransactions of the Association for Computational Linguistics, 2020
- Pre-trained models for natural language processing: A surveyScience China Technological Sciences, 2020