Dasaem Jeong
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Dasaem Jeong has a total of 46 co-authors. Here are the top co-authors by number of shared publications:
Danbinaerin Han
6 shared publications- Juhan Nam
Graduate School of Culture Technology KAIST Daejeon South Korea
4 shared publications - Xavier Serra
Universitat Pompeu Fabra, Barcelona, Spain
4 shared publications - Jose J. Valero-Mas
University of Alicante
3 shared publications - Pedro Ramoneda
Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain
3 shared publications - Danbinaerin Han
Graduate School of Culture Technology, KAIST, Daejeon, the Republic of Korea
2 shared publications
Latest Publications
- Open Access
- 6 April 2026
- Published by Ubiquity Press, Ltd. in Transactions of the International Society for Music Information Retrieval
- Vol. 9 (1) , 86-100
Six Dragons Fly Again: A Journey of Reviving 15th‑Century Korean Court Music
Despite the expanding research on symbolic music generation, there has been limited application of this to musical repertoires beyond the West. We introduce a project that revives pieces of 15th‑century Korean court music, Chihwapyeong and Chwipunghyeong, composed upon the poem Songs of the Dragon Flying to Heaven. As one of the earliest examples of Jeongganbo—a Korean musical notation system— the extant version only consists of a rudimentary melody. Our research team was commissioned by the National Gugak Center (NGC) to transform this old melody into a performable arrangement for a six‑part ensemble. Using Jeongganbo data acquired through bespoke optical music‑recognition tools and encoded using our novel approach that strictly follows the structure of Jeongganbo, we trained models for orchestral part generation and melody infilling. The resulting machine‑transformed versions of Chihwapyeong and Chwipunghyeong were evaluated by experts and performed by the Court Music Orchestra of the NGC. Our work demonstrates that generative models can successfully be applied to traditional music with limited training data if combined with careful design. By providing details of the project’s evolution, a machine‑readable dataset of Korean court music, and a library for an interactive web demo, we hope this work will directly support further research on this specific repertoire, and, equally importantly, encourage others considering similar projects on additional musical repertoires.
- 6 April 2026
- Published by Ubiquity Press, Ltd. in Transactions of the International Society for Music Information Retrieval
- Vol. 9 (1) , 86-100
Open AccessSix Dragons Fly Again: A Journey of Reviving 15th‑Century Korean Court Music
Despite the expanding research on symbolic music generation, there has been limited application of this to musical repertoires beyond the West. We introduce a project that revives pieces of 15th‑century Korean court music, Chihwapyeong and Chwipunghyeong, composed upon the poem Songs of the Dragon Flying to Heaven. As one of the earliest examples of Jeongganbo—a Korean musical notation system— the extant version only consists of a rudimentary melody. Our research team was commissioned by the National Gugak Center (NGC) to transform this old melody into a performable arrangement for a six‑part ensemble. Using Jeongganbo data acquired through bespoke optical music‑recognition tools and encoded using our novel approach that strictly follows the structure of Jeongganbo, we trained models for orchestral part generation and melody infilling. The resulting machine‑transformed versions of Chihwapyeong and Chwipunghyeong were evaluated by experts and performed by the Court Music Orchestra of the NGC. Our work demonstrates that generative models can successfully be applied to traditional music with limited training data if combined with careful design. By providing details of the project’s evolution, a machine‑readable dataset of Korean court music, and a library for an interactive web demo, we hope this work will directly support further research on this specific repertoire, and, equally importantly, encourage others considering similar projects on additional musical repertoires.
- 26 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 1876-1891
U-MusT: A Unified Framework for Cross-Modal Translation of Score Images, Symbolic Music, and Performance Audio
Music exists in various modalities, such as score images, symbolic scores, MIDI, and audio. Translations between such modalities are established as core tasks of music information retrieval, such as automatic music transcription (audio-to-MIDI) and optical music recognition (score image to symbolic score). However, most past work on multimodal translation utilizes specialized models trained for each translation task. In this paper, we propose a unified framework based on a common tokenization strategy. We use dedicated separate models for the Image-to-Audio and Audio-to-Image directions, sharing an identical encoder-decoder architecture to handle each task within a coherent framework. Two key factors make this unified approach viable: a new large-scale dataset, and the tokenization of each modality. Firstly, we propose a new dataset that consists of more than 1,300 hours of paired audio-score image data collected from YouTube videos, which is an order of magnitude larger than any existing music modal translation datasets. Secondly, our unified tokenization framework discretizes score images, audio, MIDI, and MusicXML into a sequence of tokens, enabling standard encoder-decoder Transformers to tackle multiple crossmodal translation as one coherent sequence-to-sequence task. Experimental results confirm that our unified framework improves upon single-task baselines in several key areas, notably reducing the symbol error rate for optical music recognition from 24.58% to a state-of-the-art 13.67%, while also seeing substantial improvements across the other translation tasks. Notably, our approach achieves the first musically-coherent score-image-conditioned audio generation, marking a significant breakthrough in cross-modal music generation.
- 26 December 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 34, 1876-1891
U-MusT: A Unified Framework for Cross-Modal Translation of Score Images, Symbolic Music, and Performance Audio
Music exists in various modalities, such as score images, symbolic scores, MIDI, and audio. Translations between such modalities are established as core tasks of music information retrieval, such as automatic music transcription (audio-to-MIDI) and optical music recognition (score image to symbolic score). However, most past work on multimodal translation utilizes specialized models trained for each translation task. In this paper, we propose a unified framework based on a common tokenization strategy. We use dedicated separate models for the Image-to-Audio and Audio-to-Image directions, sharing an identical encoder-decoder architecture to handle each task within a coherent framework. Two key factors make this unified approach viable: a new large-scale dataset, and the tokenization of each modality. Firstly, we propose a new dataset that consists of more than 1,300 hours of paired audio-score image data collected from YouTube videos, which is an order of magnitude larger than any existing music modal translation datasets. Secondly, our unified tokenization framework discretizes score images, audio, MIDI, and MusicXML into a sequence of tokens, enabling standard encoder-decoder Transformers to tackle multiple crossmodal translation as one coherent sequence-to-sequence task. Experimental results confirm that our unified framework improves upon single-task baselines in several key areas, notably reducing the symbol error rate for optical music recognition from 24.58% to a state-of-the-art 13.67%, while also seeing substantial improvements across the other translation tasks. Notably, our approach achieves the first musically-coherent score-image-conditioned audio generation, marking a significant breakthrough in cross-modal music generation.
- Open Access
On the de-duplication of the Lakh MIDI dataset
A large-scale dataset is essential for training a well-generalized deep-learning model. Most such datasets are collected via scraping from various internet sources, inevitably introducing duplicated data. In the symbolic music domain, these duplicates often come from multiple user arrangements and metadata changes after simple editing. However, despite critical issues such as unreliable training evaluation from data leakage during random splitting, dataset duplication has not been extensively addressed in the MIR community. This study investigates the dataset duplication issues regarding Lakh MIDI Dataset (LMD), one of the largest publicly available sources in the symbolic music domain. To find and evaluate the best retrieval method for duplicated data, we employed the Clean MIDI subset of the LMD as a benchmark test set, in which different versions of the same songs are grouped together. We first evaluated rule-based approaches and previous symbolic music retrieval models for de-duplication and also investigated with a contrastive learning-based BERT model with various augmentations to find duplicate files. As a result, we propose three different versions of the filtered list of LMD, which filters out at least 38,134 samples in the most conservative settings among 178,561 files.
Open AccessOn the de-duplication of the Lakh MIDI dataset
A large-scale dataset is essential for training a well-generalized deep-learning model. Most such datasets are collected via scraping from various internet sources, inevitably introducing duplicated data. In the symbolic music domain, these duplicates often come from multiple user arrangements and metadata changes after simple editing. However, despite critical issues such as unreliable training evaluation from data leakage during random splitting, dataset duplication has not been extensively addressed in the MIR community. This study investigates the dataset duplication issues regarding Lakh MIDI Dataset (LMD), one of the largest publicly available sources in the symbolic music domain. To find and evaluate the best retrieval method for duplicated data, we employed the Clean MIDI subset of the LMD as a benchmark test set, in which different versions of the same songs are grouped together. We first evaluated rule-based approaches and previous symbolic music retrieval models for de-duplication and also investigated with a contrastive learning-based BERT model with various augmentations to find duplicate files. As a result, we propose three different versions of the filtered list of LMD, which filters out at least 38,134 samples in the most conservative settings among 178,561 files.
- Open Access
- 9 September 2025
- Published by Association for Computing Machinery (ACM) in Journal on Computing and Cultural Heritage
- Vol. 18 (3) , 1-21
On the Automatic Recognition of Jeongganbo Music Notation: Dataset and Approach
The Jeongganbo notation, the first music representation system in East Asia capable of jointly expressing pitch and duration, has been extensively used—and still is—in the Korean music tradition since its inception in the 15th century. In this regard, there exists a plethora of music works that exclusively endure as physical sheets, which not only constitutes a heritage preservation challenge due to the inherent degradation of this format but also impedes the use of computational tools to study and exploit this music tradition. While the Optical Music Recognition (OMR) field, which represents the research area devoted to devising methods capable of automatically transcribing music sheets into digital formats, has addressed this issue in a number of music notations from the Western tradition, no previous research has considered the preservation of Jeongganbo scores. In this context, this work presents the following contributions: (i) the first data assortment of real Jeongganbo scores for OMR tasks; (ii) a collection of synthetic data generation and augmentation mechanisms to alleviate the scarcity of manual annotation; and (iii) a neural-based transcription scheme based on state-of-the-art OMR strategies specifically tailored to Jeongganbo scores. The experiments performed prove the validity of the approach—performance rates close to a 90% of success—and open new research avenues for under-resourced yet challenging music notations.
- 9 September 2025
- Published by Association for Computing Machinery (ACM) in Journal on Computing and Cultural Heritage
- Vol. 18 (3) , 1-21
Open AccessOn the Automatic Recognition of Jeongganbo Music Notation: Dataset and Approach
The Jeongganbo notation, the first music representation system in East Asia capable of jointly expressing pitch and duration, has been extensively used—and still is—in the Korean music tradition since its inception in the 15th century. In this regard, there exists a plethora of music works that exclusively endure as physical sheets, which not only constitutes a heritage preservation challenge due to the inherent degradation of this format but also impedes the use of computational tools to study and exploit this music tradition. While the Optical Music Recognition (OMR) field, which represents the research area devoted to devising methods capable of automatically transcribing music sheets into digital formats, has addressed this issue in a number of music notations from the Western tradition, no previous research has considered the preservation of Jeongganbo scores. In this context, this work presents the following contributions: (i) the first data assortment of real Jeongganbo scores for OMR tasks; (ii) a collection of synthetic data generation and augmentation mechanisms to alleviate the scarcity of manual annotation; and (iii) a neural-based transcription scheme based on state-of-the-art OMR strategies specifically tailored to Jeongganbo scores. The experiments performed prove the validity of the approach—performance rates close to a 90% of success—and open new research avenues for under-resourced yet challenging music notations.
- Open Access
Motive-level Analysis of Form-functions Association in Korean Folk song
Computational analysis of folk song audio is challenging due to structural irregularities and the need for manual annotation. We propose a method for automatic motive segmentation in Korean folk songs by fine-tuning a speech transcription model on audio lyric with motif boundary annotation. Applying this to 856 songs, we extracted motif count and duration entropy as structural features. Statistical analysis revealed that these features vary systematically according to the social function of the songs. Songs associated with collective labor, for instance, showed different structural patterns from those for entertainment or personal settings. This work offers a scalable approach for quantitative structural analysis of oral music traditions.
Open AccessMotive-level Analysis of Form-functions Association in Korean Folk song
Computational analysis of folk song audio is challenging due to structural irregularities and the need for manual annotation. We propose a method for automatic motive segmentation in Korean folk songs by fine-tuning a speech transcription model on audio lyric with motif boundary annotation. Applying this to 856 songs, we extracted motif count and duration entropy as structural features. Statistical analysis revealed that these features vary systematically according to the social function of the songs. Songs associated with collective labor, for instance, showed different structural patterns from those for entertainment or personal settings. This work offers a scalable approach for quantitative structural analysis of oral music traditions.
- 6 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2377-2387
Musical Word Embedding for Music Tagging and Retrieval
Word embedding has become an essential means for text-based information retrieval. Typically, word embeddings are learned from large quantities of general and unstructured text data. However, in the domain of music, word embedding may have difficulty understanding musical contexts or recognizing music-related entities like artists and tracks. To address this issue, we propose a new approach called Musical Word Embedding (MWE), which involves learning from various types of texts, including both everyday and music-related vocabulary. We integrate MWE into an audio-word joint representation framework for tagging and retrieving music, using words like tag, artist, and track that have different levels of musical specificity. Through extensive experiments, we demonstrate that the effectiveness of musical supervision varies by task - specifically, tag-level supervision improves tagging performance while track-level supervision enhances retrieval performance. This finding suggests that the choice of musical supervision in representation learning needs to be carefully considered based on the target task. To balance this compromise, we suggest multi-prototype training that uses words with different levels of musical specificity jointly. We evaluate both word embedding and audio-word joint embedding on four tasks (tag rank prediction, music tagging, query-by-tag, and query-by-track) across two datasets (Million Song Dataset and MTG-Jamendo). The results show that the suggested MWE is more efficient and effective for both in-domain and out-of-domain datasets compared to conventional word embedding.
- 6 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2377-2387
Musical Word Embedding for Music Tagging and Retrieval
Word embedding has become an essential means for text-based information retrieval. Typically, word embeddings are learned from large quantities of general and unstructured text data. However, in the domain of music, word embedding may have difficulty understanding musical contexts or recognizing music-related entities like artists and tracks. To address this issue, we propose a new approach called Musical Word Embedding (MWE), which involves learning from various types of texts, including both everyday and music-related vocabulary. We integrate MWE into an audio-word joint representation framework for tagging and retrieving music, using words like tag, artist, and track that have different levels of musical specificity. Through extensive experiments, we demonstrate that the effectiveness of musical supervision varies by task - specifically, tag-level supervision improves tagging performance while track-level supervision enhances retrieval performance. This finding suggests that the choice of musical supervision in representation learning needs to be carefully considered based on the target task. To balance this compromise, we suggest multi-prototype training that uses words with different levels of musical specificity jointly. We evaluate both word embedding and audio-word joint embedding on four tasks (tag rank prediction, music tagging, query-by-tag, and query-by-track) across two datasets (Million Song Dataset and MTG-Jamendo). The results show that the suggested MWE is more efficient and effective for both in-domain and out-of-domain datasets compared to conventional word embedding.
- Open Access
Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio
Music exists in various modalities, such as score images, symbolic scores, MIDI, and audio. Translations between each modality are established as core tasks of music information retrieval, such as automatic music transcription (audio-to-MIDI) and optical music recognition (score image to symbolic score). However, most past work on multimodal translation trains specialized models on individual translation tasks. In this paper, we propose a unified approach, where we train a general-purpose model on many translation tasks simultaneously. Two key factors make this unified approach viable: a new large-scale dataset and the tokenization of each modality. Firstly, we propose a new dataset that consists of more than 1,300 hours of paired audio-score image data collected from YouTube videos, which is an order of magnitude larger than any existing music modal translation datasets. Secondly, our unified tokenization framework discretizes score images, audio, MIDI, and MusicXML into a sequence of tokens, enabling a single encoder-decoder Transformer to tackle multiple cross-modal translation as one coherent sequence-to-sequence task. Experimental results confirm that our unified multitask model improves upon single-task baselines in several key areas, notably reducing the symbol error rate for optical music recognition from 24.58% to a state-of-the-art 13.67%, while similarly substantial improvements are observed across the other translation tasks. Notably, our approach achieves the first successful score-image-conditioned audio generation, marking a significant breakthrough in cross-modal music generation.
Open AccessUnified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio
Music exists in various modalities, such as score images, symbolic scores, MIDI, and audio. Translations between each modality are established as core tasks of music information retrieval, such as automatic music transcription (audio-to-MIDI) and optical music recognition (score image to symbolic score). However, most past work on multimodal translation trains specialized models on individual translation tasks. In this paper, we propose a unified approach, where we train a general-purpose model on many translation tasks simultaneously. Two key factors make this unified approach viable: a new large-scale dataset and the tokenization of each modality. Firstly, we propose a new dataset that consists of more than 1,300 hours of paired audio-score image data collected from YouTube videos, which is an order of magnitude larger than any existing music modal translation datasets. Secondly, our unified tokenization framework discretizes score images, audio, MIDI, and MusicXML into a sequence of tokens, enabling a single encoder-decoder Transformer to tackle multiple cross-modal translation as one coherent sequence-to-sequence task. Experimental results confirm that our unified multitask model improves upon single-task baselines in several key areas, notably reducing the symbol error rate for optical music recognition from 24.58% to a state-of-the-art 13.67%, while similarly substantial improvements are observed across the other translation tasks. Notably, our approach achieves the first successful score-image-conditioned audio generation, marking a significant breakthrough in cross-modal music generation.
- Open Access
LAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2
This paper introduces LAV (Latent Audio-Visual), a system that integrates EnCodec's neural audio compression with StyleGAN2's generative capabilities to produce visually dynamic outputs driven by pre-recorded audio. Unlike previous works that rely on explicit feature mappings, LAV uses EnCodec embeddings as latent representations, directly transformed into StyleGAN2's style latent space via randomly initialized linear mapping. This approach preserves semantic richness in the transformation, enabling nuanced and semantically coherent audio-visual translations. The framework demonstrates the potential of using pretrained audio compression models for artistic and computational applications.
Open AccessLAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2
This paper introduces LAV (Latent Audio-Visual), a system that integrates EnCodec's neural audio compression with StyleGAN2's generative capabilities to produce visually dynamic outputs driven by pre-recorded audio. Unlike previous works that rely on explicit feature mappings, LAV uses EnCodec embeddings as latent representations, directly transformed into StyleGAN2's style latent space via randomly initialized linear mapping. This approach preserves semantic richness in the transformation, enabling nuanced and semantically coherent audio-visual translations. The framework demonstrates the potential of using pretrained audio compression models for artistic and computational applications.
- Open Access
Boundary Regression for Leitmotif Detection in Music Audio
Leitmotifs are musical phrases that are reprised in various forms throughout a piece. Due to diverse variations and instrumentation, detecting the occurrence of leitmotifs from audio recordings is a highly challenging task. Leitmotif detection may be handled as a subcategory of audio event detection, where leitmotif activity is predicted at the frame level. However, as leitmotifs embody distinct, coherent musical structures, a more holistic approach akin to bounding box regression in visual object detection can be helpful. This method captures the entirety of a motif rather than fragmenting it into individual frames, thereby preserving its musical integrity and producing more useful predictions. We present our experimental results on tackling leitmotif detection as a boundary regression task.
Open AccessBoundary Regression for Leitmotif Detection in Music Audio
Leitmotifs are musical phrases that are reprised in various forms throughout a piece. Due to diverse variations and instrumentation, detecting the occurrence of leitmotifs from audio recordings is a highly challenging task. Leitmotif detection may be handled as a subcategory of audio event detection, where leitmotif activity is predicted at the frame level. However, as leitmotifs embody distinct, coherent musical structures, a more holistic approach akin to bounding box regression in visual object detection can be helpful. This method captures the entirety of a motif rather than fragmenting it into individual frames, thereby preserving its musical integrity and producing more useful predictions. We present our experimental results on tackling leitmotif detection as a boundary regression task.
- 7 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1129-1141
Can Audio Reveal Music Performance Difficulty? Insights From the Piano Syllabus Dataset
Automatically estimating the performance difficulty of a music piece represents a key process in music education to create tailored curricula according to the individual needs of the students. Given its relevance, the Music Information Retrieval (MIR) field comprises some proof-of-concept works addressing this task that mainly focus on high-level music abstractions such as machine-readable scores or music sheet images. In this regard, the potential of directly analyzing audio recordings has generally been neglected. This work addresses this gap in the field with two contributions: (i) PSyllabus, the first audio-based difficulty estimation dataset—collected from Piano Syllabus community—featuring 7,901 piano pieces across 11 difficulty levels from 1,233 composers as well as two additional benchmark datasets particularly compiled for evaluation purposes; and (ii) a recognition framework capable of managing different input representations—both in unimodal and multimodal manners—derived from audio to perform the difficulty estimation task. The comprehensive experimentation comprising different pre-training schemes, input modalities, and multi-task scenarios proves the validity of the hypothesis and establishes PSyllabus as a reference dataset for audio-based difficulty estimation in the MIR field. The dataset, developed code, and trained models are publicly shared to promote further research in the field.
- 7 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1129-1141
Can Audio Reveal Music Performance Difficulty? Insights From the Piano Syllabus Dataset
Automatically estimating the performance difficulty of a music piece represents a key process in music education to create tailored curricula according to the individual needs of the students. Given its relevance, the Music Information Retrieval (MIR) field comprises some proof-of-concept works addressing this task that mainly focus on high-level music abstractions such as machine-readable scores or music sheet images. In this regard, the potential of directly analyzing audio recordings has generally been neglected. This work addresses this gap in the field with two contributions: (i) PSyllabus, the first audio-based difficulty estimation dataset—collected from Piano Syllabus community—featuring 7,901 piano pieces across 11 difficulty levels from 1,233 composers as well as two additional benchmark datasets particularly compiled for evaluation purposes; and (ii) a recognition framework capable of managing different input representations—both in unimodal and multimodal manners—derived from audio to perform the difficulty estimation task. The comprehensive experimentation comprising different pre-training schemes, input modalities, and multi-task scenarios proves the validity of the hypothesis and establishes PSyllabus as a reference dataset for audio-based difficulty estimation in the MIR field. The dataset, developed code, and trained models are publicly shared to promote further research in the field.