Multi-modal Machine Learning in Engineering Design: A Review and Future Directions
Preprint
Abstract
In the rapidly advancing field of multi-modal machine learning (MMML), the convergence of multiple data modalities has the potential to reshape various applications. This paper presents a comprehensive overview of the current state, advancements, and challenges of MMML within the sphere of engineering design. The review begins with a deep dive into five fundamental concepts of MMML:multi-modal information representation, fusion, alignment, translation, and co-learning. Following this, we explore the cutting-edge applications of MMML, placing a particular emphasis on tasks pertinent to engineering design, such as cross-modal synthesis, multi-modal prediction, and cross-modal information retrieval. Through this comprehensive overview, we highlight the inherent challenges in adopting MMML in engineering design, and proffer potential directions for future research. To spur on the continued evolution of MMML in engineering design, we advocate for concentrated efforts to construct extensive multi-modal design datasets, develop effective data-driven MMML techniques tailored to design applications, and enhance the scalability and interpretability of MMML models. MMML models, as the next generation of intelligent design tools, hold a promising future to impact how products are designed.Keywords
All Related Versions
This publication has 76 references indexed in Scilit:
- TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation2024
- Cross-modal 3D Shape Generation and Manipulation2022
- Image-Based CLIP-Guided Essence Transfer2022
- VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance2022
- CentralNet: A Multilayer Approach for Multimodal Fusion2019
- Effective feature learning and fusion of multimodality data using stage‐wise deep neural network for dementia diagnosisHuman Brain Mapping, 2018
- Exploring the Limits of Weakly Supervised Pretraining2018
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images2018
- Learning Visual Features from Large Weakly Supervised Data2016
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering2016