The State of the Art Ten Years After a State of the Art: Future Research in Music Information Retrieval
Author:
- 3 April 2014
- journal article
- Published by Taylor & Francis in Journal of New Music Research
- Vol. 43 (2) , 147-172
Abstract
The GTZAN dataset appears in at least 100 published works, and is the most-used public dataset for evaluation in machine listening research for music genre recognition (MGR). Our recent work, however, shows GTZAN has several faults (repetitions, mislabelings, and distortions), which challenge the interpretability of any result derived using it. In this article, we disprove the claims that all MGR systems are affected in the same ways by these faults, and that the performances of MGR systems in GTZAN are still meaningfully comparable since they all face the same faults. We identify and analyze the contents of GTZAN, and provide a catalog of its faults. We review how GTZAN has been used in MGR research, and find few indications that its faults have been known and considered. Finally, we rigorously study the effects of its faults on evaluating five different MGR systems. The lesson is not to banish GTZAN, but to use it with consideration of its contents.Comment: 29 pages, 7 figures, 6 tables, 128 referenceKeywords
All Related Versions
This publication has 55 references indexed in Scilit:
- Scientific knowledge is possible with small-sample classificationEURASIP Journal on Bioinformatics and Systems Biology, 2013
- Zipf’s Law, Power Laws, and Music Aesthetics2011
- Effects of Album and Artist Filters in Audio Similarity Computed for Very Large Music DatabasesComputer Music Journal, 2010
- Introduction–From Genres to Tags: A Little Epistemology of Music Information Retrieval ResearchJournal of New Music Research, 2008
- Scanning the Dial: The Rapid Recognition of Music GenresJournal of New Music Research, 2008
- Autotagger: A Model for Predicting Social Tags from Acoustic Features on Large Music DatabasesJournal of New Music Research, 2008
- The music information retrieval evaluation exchange (2005-2007): A window into music information retrieval researchAcoustical Science and Technology, 2008
- Deconstructing Statistical QuestionsJournal of the Royal Statistical Society Series A: Statistics in Society, 1994
- ACM president's letter: performance analysis: experimental computer science as its bestCommunications of the ACM, 1981
- Errors of the Third Kind in Statistical ConsultingJournal of the American Statistical Association, 1957