From Metrics to Meaning: Time to Rethink Evaluation in Human–AI Collaborative Design
Open Access
- 9 January 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Interactive Intelligent Systems
- Vol. 16 (1) , 1-31
Abstract
As AI systems increasingly shape decision-making in creative design contexts, understanding how humans engage with these tools has become a critical challenge for interactive intelligent systems research. This article contributes a challenge to rethink how to evaluate human–AI collaborative systems, advocating for a more nuanced and multidimensional approach. Findings from one of the largest field studies to date ( n = 808) of a human–AI co-creative system, The Genetic Car Designer, complemented by a controlled lab study ( n = 12), are presented. The system is based on an interactive evolutionary algorithm where participants were tasked with designing a simple 2D representation of a car. Participants were exposed to galleries of design suggestions generated by an intelligent system, MAP–Elites, and a random control. Results indicate that exposure to galleries generated by MAP–Elites significantly enhanced both cognitive and behavioral engagement, leading to higher-quality design outcomes. Crucially for the wider community, the analysis reveals that conventional evaluation methods, which often focus on solely behavioral and design quality metrics, fail to capture the full spectrum of user engagement. By considering the human–AI design process as a changing emotional, behavioral, and cognitive state of the designer, we propose evaluating human–AI systems holistically and considering intelligent systems as a core part of the user experience—not simply a back-end tool.Keywords
All Related Versions
Funding Information
- EPSRC (EPSRC:EP/S021892/1)
This publication has 32 references indexed in Scilit:
- GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback2020
- Level Graph – Incremental Procedural Generation of Indoor Levels using Minimum Spanning Trees2019
- Fostering creativity in the mixed-initiative evolutionary dungeon designer2018
- Explainable AI for Designers: A Human-Centered Perspective on Mixed-Initiative Co-Creation2018
- Interactive Guidance Techniques for Improving Creative Feedback2018
- Mixed-initiative procedural generation of dungeons using game design patterns2017
- Confronting the Challenge of Quality Diversity2015
- On the visualisation of detrital age distributionsChemical Geology, 2012
- Explaining the user experience of recommender systemsUser Modeling and User-Adapted Interaction, 2012
- Designing with interactive example galleries2010