Banu Diri

Large language models (LLMs) are increasingly deployed in multilingual settings, yet their safety behavior under Turkish harmful prompts and prompt injection attempts remains insufficiently characterized. This study evaluates the adversarial robustness of 55 open- and closed-source LLMs under paired Turkish and English harmful prompt conditions. We constructed a benchmark of 790 Turkish adversarial prompts, translated the prompts into English for cross-lingual comparison, and applied both prompt sets to the model pool. Model responses were labeled as harmful, harmless, or hallucinatory, and safety was analyzed using safety scores, Turkish–English ranking differences, and inter-rater reliability based on Fleiss’ kappa. The results reveal substantial variation across models. Closed-source systems generally achieved higher safety scores and stronger filtering behavior, whereas open-source and Turkish-oriented models showed a wider performance distribution. GPT-5.4 ranked first in the Turkish tests with a 99.37% safety score but decreased to 96.71% in the English tests, while Qwen3.5:27B ranked first in English with 97.47%. These differences suggest that safety mechanisms are not fully language-invariant. Hallucination also emerged as a distinct safety risk, particularly in Turkish evaluations. The findings indicate that Turkish LLM safety cannot be inferred from general model capability alone and should be assessed through language-specific, culturally aware, and continuously updated adversarial benchmarks.

This study presents a novel empirical methodology to characterize and compare the ranking environments of major information retrieval systems, specifically Google and Bing. By analyzing technical and content attributes from a dataset of 14,465 Search Engine Results Page (SERP) items collected from a homogeneous commercial discount domain comprising 500 queries, we aim to characterize observable associative patterns between resource attributes and ranking outcomes. The dataset includes Lighthouse performance metrics and advanced content features, such as Sentence-BERT-based semantic similarity. Using K-Means clustering, we identify five resource profiles representing emergent optimization archetypes. The analysis revealed that content-related factors had a higher aggregate importance for both systems (Google: 70.1%, Bing: 61.8%) than technical factors. Specifically, Random Forest feature importance analysis highlighted that for Bing, content volume was a dominant predictor, whereas for Google, semantic relevance signals outweighed pure keyword targeting. We further contextualize these findings within an "Authority–Optimization Trade-off" framework, suggesting that Google’s negative associations for certain on-page optimization signals likely reflect a ranking function that heavily weights latent domain authority over explicit on-page compliance. These findings highlight how modern learning-to-rank systems may differentially weight explicit content features and latent authority signals when balancing relevance, diversity, and quality.

This study proposes an independent Supervisor Agent approach based on a Turkish safety classifier to protect open-source large language models against harmful prompts. The proposed TurkGuard model classifies user prompts as Safe or Unsafe before they are forwarded to the main model and provides an additional security layer by blocking harmful prompts. The study addresses large language model security threats such as Prompt Injection and Jailbreaking and discusses the limitations of existing English-oriented safety solutions due to the morphological characteristics of Turkish. Experimental results show that the fine-tuned small-scale Qwen3.5-based TurkGuard model can provide high attack detection performance and a low false alarm rate for Turkish prompts.

The development of a Turkish-specific Large Language Model (LLM) for healthcare presents a unique opportunity to enhance AI’s accessibility and relevance for Turkish-speaking medical practitioners and patients. This study introduces a specialized Turkish Medical LLM fine-tuned on over 167,732 real patient-doctor question-answer pairs sourced from a trusted medical platform and capturing authentic linguistics in Turkish medical language. Utilizing models like LLAMA 3, the fine-tuning process was supported by Low-Rank Adaptation (LoRA) and involved innovative methods to mitigate catastrophic forgetting, including spherical linear interpolation (Slerp) merging. Evaluation of the model’s performance through similarity scores, GPT-3.5 assessments, and expert reviews indicates significant improvement in the model’s ability to generate medically accurate responses. This Turkish Medical LLM demonstrates potential to support medical decision-making and patient interaction in Turkish healthcare settings, offering an essential resource for enhancing AI inclusivity across languages.

Tokenization is a fundamental preprocessing step in Natural Language Processing (NLP), significantly impacting the capability of large language models (LLMs) to capture linguistic and semantic nuances. This study introduces a novel evaluation framework addressing tokenization challenges specific to morphologically-rich and low-resource languages such as Turkish. Utilizing the Turkish MMLU (TR-MMLU) dataset, comprising 6,200 multiple-choice questions from the Turkish education system, we assessed tokenizers based on vocabulary size, token count, processing time, language-specific token percentages (\%TR), and token purity (\%Pure). These newly proposed metrics measure how effectively tokenizers preserve linguistic structures. Our analysis reveals that language-specific token percentages exhibit a stronger correlation with downstream performance (e.g., MMLU scores) than token purity. Furthermore, increasing model parameters alone does not necessarily enhance linguistic performance, underscoring the importance of tailored, language-specific tokenization methods. The proposed framework establishes robust and practical tokenization standards for morphologically complex languages.

Language models have made significant advancements in understanding and generating human language, achieving remarkable success in various applications. However, evaluating these models remains a challenge, particularly for resource-limited languages like Turkish. To address this issue, we introduce the Turkish MMLU (TR-MMLU) benchmark, a comprehensive evaluation framework designed to assess the linguistic and conceptual capabilities of large language models (LLMs) in Turkish. TR-MMLU is based on a meticulously curated dataset comprising 6,200 multiple-choice questions across 62 sections within the Turkish education system. This benchmark provides a standard framework for Turkish NLP research, enabling detailed analyses of LLMs' capabilities in processing Turkish text. In this study, we evaluated state-of-the-art LLMs on TR-MMLU, highlighting areas for improvement in model design. TR-MMLU sets a new standard for advancing Turkish NLP research and inspiring future innovations.

Bu çalışma, Türkçe sağlık danışmanlığında kullanılan dört farklı büyük dil modelinin (doktor-meta-llama-3-8b, doktor-LLama2-sambanovasystems-7b, doktor-Mistral-trendyol-7b ve doktor-llama-3-cosmos-8b) performansını değerlendirmektedir. Modeller, 321.179 hasta-doktor soru-cevap çiftinden oluşan Patient Doctor Q&A TR 321179 veri kümesi üzerinde ince ayar yapılarak eğitilmiştir. Performans ölçümünde BLEU ve BERT skor gibi sentetik değerlendirmelerin yanı sıra, Elo puanlaması ile uzman doktorların yanıt kalitesi incelemeleri de kullanılmıştır. Sonuçlar, doktor-LLama2-sambanovasystems-7b modelinin genel başarı bakımından en iyi performansı sergilediğini göstermiş, bu model uzman doktor incelemelerinden de 3.25 puan almıştır. Öte yandan, doktor-Mistral-trendyol-7b modeli %18,4 ile en düşük zararlı yanıt oranına sahip model olarak öne çıkmıştır. Bu çalışma, Türkçe sağlık hizmetlerinde yapay zekâ destekli sanal doktor asistanlarının potansiyelini göstermekte ve dile özgü modellerin geliştirilmesinin önemini vurgulamaktadır.

Today, the use of social media has become quite widespread. Among social media platforms, Twitter, now known as X, stands out with its number of users and abundance of data. This data can be used in many studies. In this study, it is aimed to predict occupation based on Turkish tweets. In the study, 5 datasets of different sizes were used. The tweets are evaluated and compared as single and pairwise. In the pre-processing step, different machine learning and deep learning methods and pre-trained models were tested using 2 different natural language processing libraries. Among the machine learning methods, the highest accuracy of 88% was obtained from the Logistic Regression model with pairwise tweet data, while the highest accuracy of 88% was obtained with the Multi-layer Perceptron from deep learning models. The BERT and "ytu-ce-cosmos/turkish-base-bert-uncased" developed by Yıldız Technical University COSMOS AI Research Team were used as pre-trained models. Although these models gave different results on different datasets, both of them achieved the highest success with a ratio of 89% on pairwise tweet data.

In recent years, the importance of computer security has increased due to the rapid advancement of digital technology, widespread Internet use, and increased sophistication of cyberattacks. Machine learning has gained great interest in securing data systems because it offers the capability of automatically detecting and responding to security threats in real time, which is crucial for maintaining the security of computer systems and protecting data from malicious attacks. This study concentrates on phishing attack detection systems, a prevalent cyber-threat. These systems assess the features of the incoming requests to identify whether they are malicious or not. Although the number of features is increasing in these systems, feature selection has become an essential pre-processing phase that identifies the most important features of a set of available features to prevent overfitting problems, improve model performance, reduce computational cost, and decrease training and execution time. Leveraging genetic algorithms, known for simulating natural selection to identify optimal solutions, we propose a novel feature selection method, based on genetic algorithms and locally optimized, that is applied to a URL-based phishing detection system with machine learning models. Our research demonstrates that the proposed technique offers a promising strategy for improving the performance of machine learning models.