Finding the Unicorn within a Million Patches: A Material-Driven Journey into the Hidden Layers of a Diffusion Model
Open Access
- 13 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 594-607
Abstract
Today's AI models learn rich internal representations, such as the visual features inside diffusion models that produce generated images, and offer a new kind of material for co-creation. However, interfaces for creating with generative AI typically operate on the level of inputs and outputs, obscuring the material formation that unfolds in between. To address this gap, this pictorial documents a material-driven journey of designing an interface for interacting with the hidden layers of diffusion models. Working in an interdisciplinary team of interpretability researchers and interaction designers, we explore how methods from mechanistic interpretability can inform alternative strategies that move beyond black-boxed interaction paradigms. Through visual exploration of technical material characteristics combining experimentation and drawing, we articulate AI's internal representations as design material. Rather than hiding model properties behind conversational agents, we present a material-driven journey for building interfaces that exposes and engages with them. Finally, we discuss how this approach might enable different forms of co-creation with generative models.Keywords
This publication has 22 references indexed in Scilit:
- Weaving Textile-form Interfaces: A Material-Driven Design Journey2023
- Examining the Text-to-Image Community of Practice: Why and How do People Prompt Generative AIs?2023
- The Prompt Artists2023
- Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works2023
- Drawing Conversations Mediated by AI2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry2021
- UX Design Innovation2017
- Foundations of Materials Experience2015
- Gestalt Theory and Computer Vision2006