An Accidental Benchmark: The History, Contingent Power, and Lasting Traces of the GTZAN Dataset
Open Access
- 2 May 2025
- journal article
- research article
- Published by Springer Nature in Digital Society
- Vol. 4 (2) , 1-21
Abstract
In 2002, George Tzanetakis presented a paper on how researchers could automatically classify musical genre from audio signals. Claiming that his model worked as well as human classifiers, Tzanetakis made his dataset available to anyone who asked for it. Ten years later, a systematic review found that this dataset had circulated on a massive scale — nearly 25% of papers on Music Genre Recognition (MGR) used the so-called GTZAN dataset in their research. Yet an analysis of the dataset revealed significant problems: repetitions, overrepresentations, files distorted to the point of corruption, with few researchers indicating that they ever listened to the musical files within it. These warnings went unheeded: the GTZAN dataset remains the most widely used dataset for MGR today. In this paper, I examine the GTZAN dataset from a historical and musicological perspective. I trace the dataset’s introduction into the Music Information Retrieval (MIR) community, and show how MIR researchers’ tendency to view the digital musical object as a set of extracted statistical features, and musical genre as a static, query-able combination of those features, created so-called ground truths about music that remain embedded in our present-day digital infrastructures. I argue that tracing this dataset’s history and ascendence to benchmark status can recover the ground-truthing process used by early MIR researchers. In addition, this history provides context for the music industry’s shift towards descriptive audio tagging and context-based recommendations.Keywords
This publication has 56 references indexed in Scilit:
- Musical Genre and Romantic Ideology2022
- Streaming Culture2021
- Genre2015
- A Survey of Evaluation in Music Genre Recognition2014
- Genre and poetics2012
- A Perceptual Study on Music Segmentation and Genre ClassificationJournal of New Music Research, 2012
- Scanning the Dial: The Rapid Recognition of Music GenresJournal of New Music Research, 2008
- Memory-Based Models of Melodic Analysis: Challenging the Gestalt PrinciplesJournal of New Music Research, 2002
- Performing Rites1996
- The Law of GenreCritical Inquiry, 1980