Comparative Analysis of ChatGPT-4 and LLaMA: Performance Evaluation on Text Summarization, Data Analysis, and Question Answering
Authors:
- 24 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-7
Abstract
ChatGPT has demonstrated the strong performance of large language models (LLMs) in natural language tasks since its November 2022 rollout by using vast datasets with trillions of parameters for training. This paper compares ChatGPT-4 and LLaMA on three NLP topics: text summarization, data analysis, and question answering using CNN/DailyMail, Airbnb Activity, and SQuAD as some of the datasets used. ChatGPT-4 outperforms LLaMA in all aspects, giving better accuracy, coherence, and relevance results, albeit with slightly higher computation costs. ChatGPT-4 performs well regarding text summarization, resulting from high BLEU scores and human evaluations for coherence, relevance, and readability. Data analyzed by ChatGPT-4 will give a more accurate insight into this topic and provide higher-quality visualizations and more precise outputs. For question answering it ranks above LLaMA in terms of precision, recall, F1 scores, correctness, and relevancy according to human ratings.Keywords
This publication has 18 references indexed in Scilit:
- CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering2024
- A Comparative Study on Annotation Quality of Crowdsourcing and LLm Via Label Aggregation2024
- Performance and Consistency of ChatGPT‐4 Versus Otolaryngologists: A Clinical Case SeriesOtolaryngology–Head and Neck Surgery, 2024
- Interpretable extractive text summarization with meta-learning and BI-LSTM: A study of meta learning and explainability techniquesExpert Systems with Applications, 2023
- Enhancing N-Gram Based Metrics with Semantics for Better Evaluation of Abstractive Text SummarizationJournal of Computer Science and Technology, 2022
- LLM: Realizing Low-Latency Memory by Exploiting Embedded Silicon Photonics for Irregular Workloads2022
- Semantic similarity metrics for evaluating source code summarization2022
- Text Summarization: A Brief Review2019
- MS-Pointer Network: Abstractive Text Summary Based on Multi-Head Self-AttentionIEEE Access, 2019
- Analysing online reviews to investigate customer behaviour in the sharing economyInformation Technology & People, 2019