A scalable framework for evaluating multiple language models through cross-domain generation and hallucination detection
Authors:
Open Access
- 16 August 2025
- journal article
- Published by Springer Nature in Scientific Reports
- Vol. 15 (1) , 29981
Abstract
Large language models (LLMs) have significantly advanced in recent years, greatly enhancing the capabilities of retrieval-augmented generation (RAG) systems. However, challenges such as semantic similarity, bias/sentiment, and hallucinations persist, especially in domain-specific applications. This paper introduces MultiLLM-Chatbot, a scalable RAG-based benchmarking framework designed to evaluate five popular LLMs GPT-4-Turbo, CLAUDE-3.7-Sonnet, LLAMA-3.3-70B, DeepSeek-R1-Zero, and Gemini-2.0-Flash across five domains: Agriculture, Biology, Economics, Internet of Things (IoT), and Medical. Fifty peer-reviewed research papers (10 per domain) were used to generate 250 standardized queries, resulting in 1,250 model responses. Texts from PDFs were extracted using PyPDF2, segmented to preserve factual coherence, embedded with sentence-transformer models, and indexed in Elasticsearch for efficient retrieval. Each response was analyzed across 4 dimensions: cosine similarity for semantic similarity, VADER sentiment analysis for sentiment detection, TF-IDF scoring, and named entity recognition (NER) for hallucination identification and factual verification. A composite scoring scheme aggregates these metrics to rank model performance. Experimental results show LLAMA-3.3-70B as the overall best-performing model, leading in all 5 domains. The proposed framework is implemented using Colab notebooks, which offer a reproducible, extensive pipeline for domain-specific LLM benchmarking. Through the combination of cross-domain analysis and multi-metric evaluation, this study fills in the gaps in current LLM benchmarking procedures and offers a modular architecture that can be adjusted to new domains and future LLM advancements. The findings inform model selection strategies for researchers and practitioners seeking trustworthy LLM deployment across diverse industrial and scientific sectors.Keywords
Funding Information
- Manipal University Jaipur
This publication has 17 references indexed in Scilit:
- Sentiment analysis in low-resource contexts: BERT’s impact on Central KurdishLanguage Resources and Evaluation, 2025
- Detecting hallucinations in large language models using semantic entropyNature, 2024
- Enhancing cloud-based large language model processing with Elasticsearch and transformer models2024
- Benchmarking Large Language Models in Retrieval-Augmented GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Holistic Evaluation of Language ModelsAnnals of the New York Academy of Sciences, 2023
- Chatbots Development Using Natural Language Processing: A Review2022
- Constructing of Semantically Dependent Patterns Based on SpaCy and StanfordNLP Libraries2021
- ColBERT2020
- Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World GraphsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media TextProceedings of the International AAAI Conference on Web and Social Media, 2014