Oriane Peter

Evidence shows that Text-to-Image (T2I) models disproportionately reflect Western cultural norms, amplifying misrepresentation and harms to minority groups. However, evaluating cultural sensitivity is inherently complex due to its fluid and multifaceted nature. This paper proposes a participatory and community-based evaluation approach to assess cultural sensitivity in T2I models, developed and validated through workshops involving 59 individuals from 19 different countries, thus embracing first-person perspectives. We describe our systematic six-step evaluation approach consisting of a reflection of evaluators’ positionality, definition of topics to be evaluated, initial reflections and expectations, image generation, quantitative and qualitative image evaluation, and final reflections. Quantitative scores and qualitative responses in our evaluation expose convergence and disagreement within and across communities, illuminate the downstream consequences of misrepresentation, and trace how training data shaped by unequal power relations distort visual depictions. The paper provides recommendations for stakeholders, highlighting pathways to investigate the sources, mechanisms, and impacts of cultural (mis)representation in T2I models.