Art or Artifice? Large Language Models and the False Promise of Creativity
Top Cited Papers
Open Access
- 11 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-34
Abstract
Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT) [64], which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose Torrance Test of Creative Writing (TTCW) to evaluate creativity as product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Collaborative Storytelling with Large-scale Neural Language Models2020
- When Design Novices and LEGO® Meet2020
- Development of Torrance test creativity thinking (TTCT) instrument in science learningAIP Conference Proceedings, 2019
- Development of a rubric for evaluating creative writing: a multi-phase researchNew Writing, 2018
- Automated Assistance for Creative Writing with an RNN Language Model2018
- WearWrite2016
- The future of crowd work2013
- Soylent2010
- Assessment of Creativity2010
- The ‘Problem’ of Creative Writing: Using Grading Rubrics Based On Narrative Theory as SolutionNew Writing, 2008