Minjun Son

This study investigates prompt engineering (PE) strategies to mitigate hallucination, a key limitation of multimodal large language models (MLLMs). To address this issue, we explore five prominent multimodal PE techniques: in-context learning (ICL), chain of thought (CoT), step-by-step reasoning (SSR), tree of thought (ToT), and retrieval-augmented generation (RAG). These techniques are systematically applied across multiple datasets with distinct domains and characteristics. Based on the empirical findings, we propose the greedy prompt engineering strategy (Greedy PES), a methodology for optimizing PE application across different datasets and MLLM models. To evaluate user satisfaction with MLLM-generated responses, we adopt a comprehensive set of evaluation metrics, including BLEU, ROUGE, METEOR, S-BERT, MoverScore, and CIDEr. A weighted aggregate evaluation score is introduced to provide a holistic assessment of model performance under varying conditions. Experimental results demonstrate that the optimal prompt engineering strategy varies significantly depending on both dataset properties and the MLLM model used. Specifically, datasets categorized as general benefit the most from ICL, ToT, and RAG, whereas mathematical datasets perform optimally with ICL, SSR, and ToT. In scientific reasoning tasks, RAG and SSR emerge as the most effective strategies. Applying Greedy PES leads to a substantial improvement in performance across different multimodal tasks, achieving an average evaluation score enhancement of 184.3% for general image captioning, 90.3% for mathematical visual question answering (VQA), and 49.1% for science visual question answering (VQA) compared to conventional approaches. These findings highlight the effectiveness of structured PE strategies in optimizing MLLM performance and provide a robust framework for PE-driven model enhancement across diverse multimodal applications.

Recent advancements in Natural Language Processing (NLP) technologies have been driven at an unprecedented pace by the development of Large Language Models (LLMs). However, challenges remain, such as generating responses that are misaligned with the intent of the question or producing incorrect answers. This paper analyzes various Prompt Engineering techniques for large-scale language models and identifies methods that can optimize response performance across different datasets without the need for extensive retraining or fine-tuning. In particular, we examine prominent Prompt Engineering techniques including In-Context Learning (ICL), Chain of Thought (CoT), Retrieval-Augmented Generation (RAG), Step-by-Step Reasoning (SSR), and Tree of Thought (ToT), and we apply these techniques to leading LLMs such as Gemma2, LlaMA3, and Mistral. The performance of these models was evaluated using the AI2 Reasoning Challenge (ARC), HellaSwag, Massive Multitask Language Understanding (MMLU), TruthfulQA, Winogrande, and Grade School Math (GSM8k) datasets across metrics such as BLEU, ROUGE, METEOR, BLEURT, and BERTScore. The experimental results indicate that the most suitable Prompt Engineering technique can vary depending on the characteristics of each dataset. Specifically, for datasets emphasizing mathematical and logical reasoning, Prompt Engineering strategies centered around CoT, SSR, and ToT were found to be advantageous. For datasets focusing on natural language understanding, ICL-centric strategies were more effective, while RAG-based strategies were beneficial for datasets where factual accuracy is crucial. However, it was also observed that the optimal combination of Prompt Engineering techniques could differ depending on the specific LLM, indicating that fine-tuning the Prompt Engineering approach to the model and dataset is essential for achieving the best performance. The findings indicate that as LLMs become more advanced, their reliance on Prompt Engineering (PE) techniques diminishes, yet the magnitude of their performance improvement when PE strategies are applied increases. Furthermore, these advanced models tend to depend less on ICL techniques while exhibiting a greater reliance on RAG strategies. It is also evident that implementing RAG with PE-based preprocessing yields superior performance enhancements compared to the mere application of RAG on raw data.