In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker
Preprint
Abstract
Recent advances in multimodal large language models (LLMs) have lowered the barriers to rapidly prototyping AI-powered features via prompting, especially for mobile-intended use cases. Despite the value of situated user feedback, the process of soliciting early, mobile-situated user feedback on AI prototypes remains challenging. The broad scope and flexibility of LLMs means that, for a given use-case-specific prototype, there is a crucial need to understand the wide range of in-the-wild input likely to be provided by the user, as well as their in-context expectations of the AI's behavior. To explore the concept of in situ AI prototyping and testing, we created MobileMaker: an AI prototyping tool that enables designers to rapidly create mobile AI prototypes that can be tested on-device, and enables testers to make on-device, in-the-field revisions of the prototype through natural language. In an exploratory study with 16 users, we explored how user feedback on prototypes created with MobileMaker compares to that of existing prototyping tools (e.g., Figma, prompt editors). We found that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI's and user's in-context interpretation of the task, and contextual signals missed by the AI. Furthermore, we learned that while the ability to make in-the-wild revisions led users to feel more fulfilled as active participants in the design process, it might also constrain their feedback to the subset of changes perceived as more actionable or implementable by the prototyping tool.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- Building Your Own Product Copilot: Challenges, Opportunities, and Needs2025
- RETRACTED: Effectiveness of embodied evaluation of mobile applications: A qualitative studyHeliyon, 2023
- The effects of prototype medium on usability testingApplied Ergonomics, 2013
- Parallel prototyping leads to better design results, more divergence, and increased self-efficacyACM Transactions on Computer-Human Interaction, 2010
- Skinput2010
- A mobile tool for in-situ prototyping2009
- The anatomy of prototypesACM Transactions on Computer-Human Interaction, 2008
- Wizard of Oz Support throughout an Iterative Design ProcessIEEE Pervasive Computing, 2005
- Low vs. high-fidelity prototyping debateInteractions, 1996
- Prototyping an intelligent agent through Wizard of Oz1993