Creation, Evaluation and Self-Validation of Simulation Models with Large Language Models
Preprint
- 2 May 2025
- preprint
- Published by Springer Nature in Research Square
Abstract
Engineering tasks are significantly underrepresented in current large language model (LLM) datasets and research, despite their complexity and practical importance. These tasks often demand a deep mathematical and involve a combination of textual descriptions, visual representations, and numerical data. Moreover, engineering frequently relies on accepted approximations and models rather than exact values. Therefore, the present paper advances the integration of LLMs into mechanical engineering by introducing a comprehensive framework for automated simulation model generation and validation. The framework is designed as a benchmark and focuses on mechanical engineering problems in dynamics in particular on multibody dynamics simulation models in Python. It allows for the creation of a large number of test cases due to its use of parametrized models with ground truth solutions, allowing evaluation for executability and correctness. Lastly, LLM-agents are employed to generate simulation models and perform self-evaluation through a predefined set of validation methods, assessing models for parametrization errors. Evaluation results using classical F-score metrics demonstrate that most tested LLMs identify a majority of incorrect models, while the best-performing model achieves high accuracy in differing between correct and wrong simulation models.Keywords
All Related Versions
- Version 1, 2025-05-02, Research Square
- Published version: Neurocomputing, 663.
This publication has 20 references indexed in Scilit:
- Large language models and linguistic intentionalitySynthese, 2024
- ChatGPT‐3.5 and ‐4.0 and mechanical engineering: Examining performance on the FE mechanical engineering and undergraduate examsComputer Applications in Engineering Education, 2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledgeExtreme Mechanics Letters, 2024
- Multibody Models Generated from Natural LanguageMultibody System Dynamics, 2024
- ChatGPT has entered the classroom: how LLMs could transform educationNature, 2023
- Exudyn – a C++-based Python package for flexible multibody systemsMultibody System Dynamics, 2023
- Measuring follow-on innovationResearch Policy, 2023
- Artificial intelligence, machine learning and deep learning in advanced robotics, a reviewCognitive Robotics, 2023
- Applications of machine learning to machine fault diagnosis: A review and roadmapMechanical Systems and Signal Processing, 2020