Performance Analysis of Prompt-Engineering Techniques for Large Language Model
Authors:
- 11 January 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
In this paper, we analyze the performance of three large language models (LLMs) - Llama3, Gemma2, and Mistral - using various combinations of prompt engineering techniques, focusing on the TruthfulQA and Winogrande datasets. In TruthfulQA, which emphasizes evaluating the truthfulness and cognitive errors of language models, the Chain of Thought (CoT) and Retrieval-Augmented Generation (RAG) techniques demonstrated superior performance. On the other hand, in Wino-grande, which assesses contextual reasoning abilities based on common-sense knowledge, CoT and In-Context Learning (ICL) were found to be effective. The performance analysis by model revealed that Llama3 exhibited the most significant improvement when prompt engineering techniques were applied. Meanwhile, Gemma2 achieved the highest overall performance across all evaluation metrics. This study highlights the importance of selecting appropriate prompt engineering techniques tailored to each model to maximize LLM performance, demonstrating that effective combinations of techniques can substantially enhance the models' reasoning abilities and accuracy.Keywords
This publication has 8 references indexed in Scilit:
- A Survey on In-context Learning2024
- Procedural Text Mining with Large Language Models2023
- Pre-Training to Learn in Context2023
- MetaICL: Learning to Learn In Context2022
- TruthfulQA: Measuring How Models Mimic Human Falsehoods2022
- WinoGrandeCommunications of the ACM, 2021
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm2021
- Text Summarization with Pretrained Encoders2019