What’s in a Name? Evaluating Assembly-Part Semantic Knowledge in Language Models Through User-Provided Names in Computer Aided Design Files
- 23 June 2023
- journal article
- research article
- Published by Wiley in Journal of Computing and Information Science in Engineering
- Vol. 24 (1) , 1-17
Abstract
Semantic knowledge of part-part and part-whole relationships in assemblies is useful for a variety of tasks from searching design repositories to the construction of engineering knowledge bases. In this work, we propose that the natural language names designers use in computer aided design (CAD) software are a valuable source of such knowledge, and that large language models (LLMs) contain useful domain-specific information for working with this data as well as other CAD and engineering-related tasks. In particular, we extract and clean a large corpus of natural language part, feature, and document names and use this to quantitatively demonstrate that a pre-trained language model can outperform numerous benchmarks on three self-supervised tasks, without ever having seen this data before. Moreover, we show that fine-tuning on the text data corpus further boosts the performance on all tasks, thus demonstrating the value of the text data which until now has been largely ignored. We also identify key limitations to using LLMs with text data alone, and our findings provide a strong motivation for further work into multi-modal text-geometry models. To aid and encourage further work in this area we make all our data and code publicly available.Keywords
All Related Versions
Funding Information
- Autodesk (N/A)
This publication has 23 references indexed in Scilit:
- A Data-Driven Text Mining and Semantic Network Analysis for Design Information RetrievalJournal of Mechanical Design, 2017
- Learning hierarchical shape segmentation and labeling from online repositoriesACM Transactions on Graphics, 2017
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015
- Biologically Meaningful Keywords for Functional Terms of the Functional BasisJournal of Mechanical Design, 2011
- Context-based search for 3D modelsACM Transactions on Graphics, 2010
- Introduction of a data schema to support a design repositoryComputer-Aided Design, 2008
- ConceptNet — A Practical Commonsense Reasoning Tool-KitBT Technology Journal, 2004
- A search engine for 3D modelsACM Transactions on Graphics, 2003
- Design repositories: engineering design's new knowledge baseIEEE Intelligent Systems and their Applications, 2000
- WordNetCommunications of the ACM, 1995