Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design
Preprint
Abstract
Deep generative models, such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown great promise in a variety of applications, including image and speech synthesis, natural language processing, and drug discovery. However, when applied to engineering design problems, evaluating the performance of these models can be challenging, as traditional statistical metrics based on likelihood may not fully capture the requirements of engineering applications. This paper doubles as a review and a practical guide to evaluation metrics for deep generative models (DGMs) in engineering design. We first summarize well-accepted `classic' evaluation metrics for deep generative models grounded in machine learning theory and typical computer science applications. Using case studies, we then highlight why these metrics seldom translate well to design problems but see frequent use due to the lack of established alternatives. Next, we curate a set of design-specific metrics which have been proposed across different research communities and can be used for evaluating deep generative models. These metrics focus on unique requirements in design and engineering, such as constraint satisfaction, functional performance, novelty, and conditioning. We structure our review and discussion as a set of practical selection criteria and usage guidelines. Throughout our discussion, we apply the metrics to models trained on simple 2-dimensional example problems. Finally, to illustrate the selection process and classic usage of the presented metrics, we evaluate three deep generative models on a multifaceted bicycle frame design problem considering performance target achievement, design novelty, and geometric constraints. We publicly release the code for the datasets, models, and metrics used throughout the paper at decode.mit.edu/projects/metrics/.Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- FRAMED: An AutoML Approach for Structural Performance Prediction of Bicycle FramesComputer-Aided Design, 2022
- On the use of artificial neural networks in topology optimisationStructural and Multidisciplinary Optimization, 2022
- Generative models for molecular discovery: Recent advances and challengesWIREs Computational Molecular Science, 2022
- MO-PaDGAN: Reparameterizing Engineering Designs for augmented multi-objective optimizationApplied Soft Computing, 2021
- Integrating deep learning into CAD/CAE system: generative design and evaluation of 3D conceptual wheelStructural and Multidisciplinary Optimization, 2021
- Evaluation Metrics for Conditional Image GenerationInternational Journal of Computer Vision, 2021
- Combining structural performance and designer preferences in evolutionary design space explorationAutomation in Construction, 2015
- Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimationAnnals of the Institute of Statistical Mathematics, 2011
- Generalized shape optimization without homogenizationStructural and Multidisciplinary Optimization, 1992
- Generating optimal topologies in structural design using a homogenization methodComputer Methods in Applied Mechanics and Engineering, 1988