Exploring the Behavior and Performance of Large Language Models: Can LLMs Infer Answers to Questions Involving Restricted Information?
Preprint
- 7 November 2024
- preprint
- Published by MDPI AG in Preprints.org
Abstract
In this paper various LLMs are tested in a specific domain using a Retrieval-Augmented Generation (RAG) system. The study focuses on the performance and behaviour of the models and was conducted in Spanish. A questionnaire based on The Bible, which consist of questions that vary in complexity of reasoning, was created in order to evaluate the reasoning capabilities of each model. The RAG system matches a question with the most similar passage from The Bible and feeds the pair to each LLM. The evaluation aims to determine whether each model can reason solely with the provided information or if it disregards the instructions given and makes use on its pretrained knowledge.Keywords
All Related Versions
- Version 1, 2024-11-07, Preprints
- Published version: Information, 16 (2), 77.
This publication has 5 references indexed in Scilit:
- MedChatZH: A tuning LLM for traditional Chinese medicine consultationsComputers in Biology and Medicine, 2024
- Benchmarking Retrieval-Augmented Generation for Medicine2024
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
- Finding Answers from the Word of God: Domain Adaptation for Neural Networks in Biblical Question Answering2018