Evaluating Large Language Models on Financial Report Summarization: An Empirical Study
Preprint
Abstract
In recent years, Large Language Models (LLMs) have demonstrated remarkable versatility across various applications, including natural language understanding, domain-specific knowledge tasks, etc. However, applying LLMs to complex, high-stakes domains like finance requires rigorous evaluation to ensure reliability, accuracy, and compliance with industry standards. To address this need, we conduct a comprehensive and comparative study on three state-of-the-art LLMs, GLM-4, Mistral-NeMo, and LLaMA3.1, focusing on their effectiveness in generating automated financial reports. Our primary motivation is to explore how these models can be harnessed within finance, a field demanding precision, contextual relevance, and robustness against erroneous or misleading information. By examining each model's capabilities, we aim to provide an insightful assessment of their strengths and limitations. Our paper offers benchmarks for financial report analysis, encompassing proposed metrics such as ROUGE-1, BERT Score, and LLM Score. We introduce an innovative evaluation framework that integrates both quantitative metrics (e.g., precision, recall) and qualitative analyses (e.g., contextual fit, consistency) to provide a holistic view of each model's output quality. Additionally, we make our financial dataset publicly available, inviting researchers and practitioners to leverage, scrutinize, and enhance our findings through broader community engagement and collaborative improvement. Our dataset is available on huggingface.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- DialogueLLM: Context and emotion knowledge-tuned large language models for emotion recognition in conversationsNeural Networks, 2025
- Pre-Trained Language Models for Text Generation: A SurveyACM Computing Surveys, 2024
- Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers2024
- AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization2024
- Large language models in law: A surveyAI Open, 2024
- Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis2024
- Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicineEBioMedicine, 2023
- FinBERT: A Large Language Model for Extracting Information from Financial Text*Contemporary Accounting Research, 2023
- Text Classification via Large Language Models2023
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019