Evaluating Large Language Models for Material Selection
- 14 November 2024
- journal article
- research article
- Published by Wiley in Journal of Computing and Information Science in Engineering
- Vol. 25 (2) , 1-33
Abstract
Material selection is a crucial step in conceptual design due to its significant impact on the functionality, aesthetics, manufacturability, and sustainability impact of the final product. This study investigates the use of large language models (LLMs) for material selection in the product design process and compares the performance of LLMs against expert choices for various design scenarios. By collecting a dataset of expert material preferences, the study provides a basis for evaluating how well LLMs can align with expert recommendations through prompt engineering and hyperparameter tuning. The divergence between LLM and expert recommendations is measured across different model configurations, prompt strategies, and temperature settings. This approach allows for a detailed analysis of factors influencing the LLMs' effectiveness in recommending materials. The results from this study highlight two failure modes: the low variance of recommendations across different design scenarios and the tendency toward overestimating material appropriateness. Parallel prompting is identified as a useful prompt-engineering method when using LLMs for material selection. The findings further suggest that, while LLMs can provide valuable assistance, their recommendations often vary significantly from those of human experts. This discrepancy underscores the need for further research into how LLMs can be better tailored to replicate expert decision-making in material selection. This work contributes to the growing body of knowledge on how LLMs can be integrated into the design process, offering insights into their current limitations and potential for future improvements.Keywords
All Related Versions
This publication has 54 references indexed in Scilit:
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Subjective selection and the evolution of complex cultureEvolutionary Anthropology: Issues, News, and Reviews, 2022
- Integrating material selection with design optimization via neural networksEngineering with Computers, 2022
- Online question and answer sessions: How students support their own and other students' processes of inquiry in a text-based learning environmentThe Internet and Higher Education, 2021
- How you choose is as important as what you choose: Subjective quality of choice predicts well-being and academic performanceCurrent Psychology, 2020
- A framework for concurrent material and process selection during conceptual product design stagesMaterials & Design (1980-2015), 2012
- A material selection methodology and expert system for sustainable product designThe International Journal of Advanced Manufacturing Technology, 2011
- Multi-objective optimization of material selection for sustainable products: Artificial neural networks and genetic algorithm approachMaterials & Design (1980-2015), 2009
- A novel method for materials selection in mechanical design: Combination of non-linear normalization and a modified digital logic methodMaterials & Design (1980-2015), 2007
- Materials selection combined with optimal structural design: concept and some resultsMaterials & Design (1980-2015), 2002