Investigating Complex Sketch Recognition Strategies for Developing Future Design Tools
Authors:
- 27 October 2025
- journal article
- Published by Wiley in Journal of Mechanical Design
- Vol. 148 (4)
Abstract
Despite recent advances in multi-modal artificial intelligence (AI) technology, there remains a significant gap in their ability to be incorporated into complex design and engineering work. One such challenge relates to contexts where sketch-based inputs are desirable, due to the difficulty in recognizing freehand sketches or interpreting underlying human intent. To elucidate requirements for emerging sketch-based AI systems for complex design contexts, we consider an architectural design case study. Using a Wizard of Oz experimental paradigm, we substitute the “tool” with human agents and conduct a lab-based study in which professional architectural designers complete a design brief using this “tool.” Here, the human agents execute functions such as recognizing freely produced design plans and perspective drawings for downstream applications (e.g., generating inspirational images or high-quality renders). Observing the human agents performing the sketch recognition task, results demonstrate that agents not only rely on visible sketch elements (i.e., lines) and architectural drawing codes, but also on their memory of previous lines and their knowledge of the design brief to comprehend perceived lines. Agents gradually develop an understanding of the designed artifact, but also of the designer's intentions. These activities are crucial for the agent to obtain a functional model of the designed object, beyond a purely topological and geometric perception model. Insights about this human workflow bring new potential techniques of sketch recognition for design and engineering tasks, informing the inclusion of new resources within AI tools.Keywords
This publication has 27 references indexed in Scilit:
- The effect of AI-based inspiration on human design ideationInternational Journal of Design Creativity and Innovation, 2023
- Data-enabled sketch search and retrieval for visual design stimuli generationArtificial Intelligence for Engineering Design, Analysis and Manufacturing, 2022
- Enabling multi-modal search for inspirational design stimuli using deep learningArtificial Intelligence for Engineering Design, Analysis and Manufacturing, 2022
- Life cycle assessment in the building design process – A systematic literature reviewBuilding and Environment, 2020
- A Hybrid convolutional neural network for sketch recognitionPattern Recognition Letters, 2020
- Advancing an understanding of design cognition and design metacognition: Progress and prospectsDesign Studies, 2019
- A survey of sketch based modeling systemsFrontiers of Computer Science, 2016
- Human and Computational Approaches for Design Problem-Solving2016
- Fixation or inspiration? A meta-analytic review of the role of examples on design processesDesign Studies, 2015
- The Electronic Cocktail Napkin—a computational environment for working with design diagramsDesign Studies, 1996