In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker
Authors:
- 2 September 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 44, 121-133
Abstract
Recent advances in multimodal large language models (LLMs) have made it easier to rapidly prototype AI-powered features, especially for mobile use cases. However, gathering early, mobile-situated user feedback on these AI prototypes remains challenging. The broad scope and flexibility of LLMs means that, for a given use-case-specific prototype, there is a crucial need to understand the wide range of in-the-wild input users are likely to provide and their in-context expectations for the AI’s behavior. To explore the concept of in situ AI prototyping and testing, we created MobileMaker: a platform that enables designers to rapidly create and test mobile AI prototypes directly on devices. This tool also enables testers to make on-device, in-the-field revisions of prototypes using natural language. In an exploratory study with 16 participants, we explored how user feedback on prototypes created with MobileMaker compares to that of existing prototyping tools (e.g., Figma, prompt editors). Our findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI’s and user’s in-context interpretation of the task, and contextual signals missed by the AI. Furthermore, we learned that while the ability to make in-the-wild revisions led users to feel more fulfilled as active participants in the design process, it might also constrain their feedback to the subset of changes perceived as more actionable or implementable by the prototyping tool.Keywords
All Related Versions
This publication has 46 references indexed in Scilit:
- RETRACTED: Effectiveness of embodied evaluation of mobile applications: A qualitative studyHeliyon, 2023
- The effects of prototype medium on usability testingApplied Ergonomics, 2013
- Change by DesignJournal of Product Innovation Management, 2011
- Skinput2010
- A mobile tool for in-situ prototyping2009
- The anatomy of prototypesACM Transactions on Computer-Human Interaction, 2008
- Wizard of Oz Support throughout an Iterative Design ProcessIEEE Pervasive Computing, 2005
- Machine learning for adaptive user interfaces1997
- Iterative user-interface designComputer, 1993
- Prototyping an intelligent agent through Wizard of Oz1993