Jigsaw: Supporting Designers in Prototyping Multimodal Applications by Assembling AI Foundation Models
Preprint
Abstract
Recent advancements in AI foundation models have made it possible for them to be utilized off-the-shelf for creative tasks, including ideating design concepts or generating visual prototypes. However, integrating these models into the creative process can be challenging as they often exist as standalone applications tailored to specific tasks. To address this challenge, we introduce Jigsaw, a prototype system that employs puzzle pieces as metaphors to represent foundation models. Jigsaw allows designers to combine different foundation model capabilities across various modalities by assembling compatible puzzle pieces. To inform the design of Jigsaw, we interviewed ten designers and distilled design goals. In a user study, we showed that Jigsaw enhanced designers' understanding of available foundation model capabilities, provided guidance on combining capabilities across different modalities and tasks, and served as a canvas to support design exploration, prototyping, and documentation.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking HeadProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- Creating Design Resources to Scaffold the Ideation of AI Concepts2023
- Measuring and Narrowing the Compositionality Gap in Language Models2023
- A configurable visual programming environment: a tool for domain-specific programmingComputer, 1995