Evaluating large language models on business process modeling: framework, benchmark, and self-improvement analysis
Top Cited Papers
Open Access
- 2 September 2025
- journal article
- research article
- Published by Springer Nature in Software and Systems Modeling
- Vol. 25 (4) , 1-36
Abstract
Large language models (LLMs) are rapidly transforming various fields, including the field of business process management (BPM). LLMs provide new ways for analyzing and improving operational processes. This paper assesses the capabilities of LLMs on business process modeling using a framework for automating this task and a robust evaluation approach. We design a comprehensive benchmark, consisting of 20 diverse business processes, and we demonstrate our evaluation approach by assessing 16 current state-of-the-art LLMs from major AI vendors. Our analysis highlights significant performance variations across LLMs and reveals a positive correlation between efficient error handling and the quality of generated models. It also shows consistent performance trends within similar LLM groups. Furthermore, we use our evaluation approach to investigate LLM self-improvement techniques, encompassing self-evaluation, input optimization, and output optimization. Our findings indicate that output optimization, in particular, offers promising potential for enhancing quality, especially in models with initially lower performance. Our contributions provide insights for leveraging LLMs in BPM, paving the way for more advanced and automated process modeling techniques.Keywords
All Related Versions
Funding Information
- Fraunhofer-Institut für Angewandte Informationstechnik FIT
This publication has 78 references indexed in Scilit:
- Large Language Models Can Accomplish Business Process Management Tasks2024
- Knowledge Injection to Counter Large Language Model (LLM) Hallucination2023
- Large Language Models for Business Process Management: Opportunities and Challenges2023
- POWL: Partially Ordered Workflow Language2023
- Conversational Process Modelling: State of the Art, Applications, and Implications in Practice2023
- PET: An Annotated Dataset for Process Extraction from Natural Language Text Tasks2023
- Conformance Checking: Foundations, Milestones and Challenges2022
- Robust Process Mining with Guarantees2022
- An Approach for Process Model Extraction by Multi-grained Text Classification2020
- A Survey of Process Mining Competitions: The BPI Challenges 2011–20182019