Olivier Lartillot
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Olivier Lartillot has a total of 33 co-authors. Here are the top co-authors by number of shared publications:
- Anna-Maria Christodoulou
RITMO Center for Interdisciplinary Studies in Rhythm, Time, and Motion, Department of Musicology, University of Oslo, Oslo, Norway
3 shared publications - Alexander Refsum Jensenius
University of Oslo
2 shared publications - Anastasiia Mazurenko
Institute of Ethnomusicology, Slovenian Academy of Science and Arts
1 shared publication - Anders Elowsson
KTH Royal Institute of Technology, School of Electrical Engineering and Computer Science , Stockholm, Sweden
1 shared publication Anne Danielsen
RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, University of Oslo , Oslo, Norway
1 shared publicationBruno Laeng
RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, University of Oslo, Oslo, Norway
1 shared publication
Latest Publications
- Open Access
- 2 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 550-560
Exploring Real-Time Visualizations as Support for Concert experiences for Hearing-Impaired
- 2 September 2026
- Published by Association for Computing Machinery (ACM)
- p. 550-560
Open AccessExploring Real-Time Visualizations as Support for Concert experiences for Hearing-Impaired
- Open Access
- 25 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 19-27
A Multimodal Dataset of Greek Folk Music
- 25 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 19-27
Open AccessA Multimodal Dataset of Greek Folk Music
- Open Access
- 23 June 2025
- Published by Center for Open Science in OSF Preprints
MusiQAl: A dataset for Music Question-Answering through Audio-Video Fusion
Music Question--Answering (MQA) is a machine learning task where a computational system analyzes and answers questions about music-related data. Traditional methods prioritize audio, overlooking visual and embodied aspects crucial to music performance understanding. We introduce MusiQAl, a multimodal dataset of 310 music performance videos and 11,793 human-annotated question--answer pairs, spanning diverse musical traditions and styles. Grounded in musicology and music psychology, MusiQAl emphasizes multimodal reasoning, causal inference, and cross-cultural understanding of performer-music interactions. We benchmark AVST and LAVISH on MusiQAl, revealing strengths and limitations, underscoring the importance of integrating multimodal learning and domain expertise to advance MQA and Music Information Retrieval (MIR).
- 23 June 2025
- Published by Center for Open Science in OSF Preprints
Open AccessMusiQAl: A dataset for Music Question-Answering through Audio-Video Fusion
Music Question--Answering (MQA) is a machine learning task where a computational system analyzes and answers questions about music-related data. Traditional methods prioritize audio, overlooking visual and embodied aspects crucial to music performance understanding. We introduce MusiQAl, a multimodal dataset of 310 music performance videos and 11,793 human-annotated question--answer pairs, spanning diverse musical traditions and styles. Grounded in musicology and music psychology, MusiQAl emphasizes multimodal reasoning, causal inference, and cross-cultural understanding of performer-music interactions. We benchmark AVST and LAVISH on MusiQAl, revealing strengths and limitations, underscoring the importance of integrating multimodal learning and domain expertise to advance MQA and Music Information Retrieval (MIR).
- Open Access
- 1 March 2025
- Published by SAGE Publications in Music & Science
- Vol. 8
Video Visualization of a String Quartet Performance of a Bach Fugue: Design and Subjective Evaluation
Visualizing music—through music notation, analytical representations, or music videos—might potentially boost the appreciation of music in all its richness. The purpose of this study was to design and test a visualization strategy aimed at explicating to a large audience with diverse backgrounds—especially novices—the multifaceted beauty of the final Contrapunctus in J.S. Bach's The Art of Fugue , performed by the Danish String Quartet. At the surface level of the musical structure, the rich fluctuation of pitch shaped by each musician was depicted in the form of undulating pitch curves. At a deeper structural level, the repetition of pitch curves, distinctive of fugues, was highlighted through vertical alignment—inspired by a technique called paradigmatic analysis, originating from anthropology and music semiology. The visualization was initially prototyped in the form of a real-time technology as part of the MusicLab Copenhagen research concert. The concert audience focused on the performance itself, and did not pay much attention to, nor appreciate, the visualization. To evaluate more thoroughly the potential of the visualization, participants with varied musical expertise and taste were invited to listen to a recorded performance of the piece and watch the visualization on their own computer. A large majority reported that they felt they understood the visualization, around half of them felt that it enhanced their musical understanding, and a small group felt that it helped them to better appreciate the music.
- 1 March 2025
- Published by SAGE Publications in Music & Science
- Vol. 8
Open AccessVideo Visualization of a String Quartet Performance of a Bach Fugue: Design and Subjective Evaluation
Visualizing music—through music notation, analytical representations, or music videos—might potentially boost the appreciation of music in all its richness. The purpose of this study was to design and test a visualization strategy aimed at explicating to a large audience with diverse backgrounds—especially novices—the multifaceted beauty of the final Contrapunctus in J.S. Bach's The Art of Fugue , performed by the Danish String Quartet. At the surface level of the musical structure, the rich fluctuation of pitch shaped by each musician was depicted in the form of undulating pitch curves. At a deeper structural level, the repetition of pitch curves, distinctive of fugues, was highlighted through vertical alignment—inspired by a technique called paradigmatic analysis, originating from anthropology and music semiology. The visualization was initially prototyped in the form of a real-time technology as part of the MusicLab Copenhagen research concert. The concert audience focused on the performance itself, and did not pay much attention to, nor appreciate, the visualization. To evaluate more thoroughly the potential of the visualization, participants with varied musical expertise and taste were invited to listen to a recorded performance of the piece and watch the visualization on their own computer. A large majority reported that they felt they understood the visualization, around half of them felt that it enhanced their musical understanding, and a small group felt that it helped them to better appreciate the music.
- Open Access
- 30 December 2024
- Published by Editorial CSIC in Anuario Musical
- No. 79,p. 560
Perspectives on digital music edition, heritage preservation, annotation and analysis: a COST Action meeting in Barcelona
This article brings together the contribution of the participants in a Short-Term Scientific Mission (STSM) of the COST Action EarlyMuse (CA21161) which took place at the IMF-CSIC in Barcelona (June 2024). The discussions involved different aspects of projects with digital music edition, annotation, analysis, and the problems related to preservation of music heritage, with particular attention to the situation in Ukraine. The diverse geographical and professional background and expertise of the authors provides, in their respective sections of the article, a rich panorama on technological projects, issues, models, interoperability, and digital tools. A final Appendix contains a useful list, with links, of the projects, websites, and tools mentioned during the STSM meeting.
- 30 December 2024
- Published by Editorial CSIC in Anuario Musical
- No. 79,p. 560
Open AccessPerspectives on digital music edition, heritage preservation, annotation and analysis: a COST Action meeting in Barcelona
This article brings together the contribution of the participants in a Short-Term Scientific Mission (STSM) of the COST Action EarlyMuse (CA21161) which took place at the IMF-CSIC in Barcelona (June 2024). The discussions involved different aspects of projects with digital music edition, annotation, analysis, and the problems related to preservation of music heritage, with particular attention to the situation in Ukraine. The diverse geographical and professional background and expertise of the authors provides, in their respective sections of the article, a rich panorama on technological projects, issues, models, interoperability, and digital tools. A final Appendix contains a useful list, with links, of the projects, websites, and tools mentioned during the STSM meeting.
- Open Access
- 27 March 2024
- Published by Center for Open Science in OSF Preprints
Multimodal Music Datasets? Challenges and Future Goals in Music Processing
The term ``multimodal music dataset'' is often used to describe music-related data that represent music as a multimedia art form and multimodal experience. However, the term ``multimodality'' may mean different things in related disciplines, such as musicology, music psychology, and music technology. This paper proposes a definition of multimodality that works across different music disciplines. Many challenges are related to constructing, evaluating, and using multimodal music datasets. We provide a task-based categorization of these datasets and explore theoretical methodologies aimed at enhancing their future construction. Diverse data preprocessing methods are illuminated, highlighting their contributions to transparent music analysis. Additionally, evaluation metrics, methods, and benchmarks tailored for multimodal music processing tasks are scrutinized, empowering researchers to make informed decisions and facilitating cross-study comparisons.
- 27 March 2024
- Published by Center for Open Science in OSF Preprints
Open AccessMultimodal Music Datasets? Challenges and Future Goals in Music Processing
The term ``multimodal music dataset'' is often used to describe music-related data that represent music as a multimedia art form and multimodal experience. However, the term ``multimodality'' may mean different things in related disciplines, such as musicology, music psychology, and music technology. This paper proposes a definition of multimodality that works across different music disciplines. Many challenges are related to constructing, evaluating, and using multimodal music datasets. We provide a task-based categorization of these datasets and explore theoretical methodologies aimed at enhancing their future construction. Diverse data preprocessing methods are illuminated, highlighting their contributions to transparent music analysis. Additionally, evaluation metrics, methods, and benchmarks tailored for multimodal music processing tasks are scrutinized, empowering researchers to make informed decisions and facilitating cross-study comparisons.
- Open Access
- 13 December 2023
- Published by Ubiquity Press, Ltd. in Transactions of the International Society for Music Information Retrieval
- Vol. 6 (1) , 186-202
A Dataset of Norwegian Hardanger Fiddle Recordings with Precise Annotation of Note and Beat Onsets
The Hardanger fiddle is a variety of the violin used in the folk music of the western and central part of southern Norway. This paper presents a dataset of several hours of recordings of Hardanger fiddle music, with note annotations of onsets, offsets and pitches, provided by the performers themselves. A subset has also been annotated with beat onset positions by the performer as well as three expert musicians. The complexity of the music genre—polyphonic, highly ornamented and with a very irregular pulsation, among other aspects—motivated the design of a new annotation software adapted to these particular needs. Beat annotation in MIR is typically recorded as positions in seconds, without explicit connection with actual musical events. In the context of music where the rhythm is carried by the melodic instrument alone, a more reliable definition of beat onsets consists in associating them with the onsets of the notes that represent the start of each beat. This latter definition of beat onsets reflects that beats are generated from within the flow of played melodic-rhythmic events, which implies that the spacing of beats may be shifting and irregular. This motivated the design of a new method for beat annotation in Hardanger fiddle music based on a selection of notes in the note annotation. Comparisons between annotators through alignment—integrated in the interface—enable them to eventually correct their annotations or observe alternative valid interpretations of any given excerpt. After dedicating a part of the note annotation dataset to the training of a machine learning model, for the task of assessing both note pitch and onset time, an F1 score of 87% can be reached. The beat annotation dataset demonstrates the necessity of developing new beat trackers adapted to Hardanger fiddle music. The dataset as well as the annotation software is made publicly available.
- 13 December 2023
- Published by Ubiquity Press, Ltd. in Transactions of the International Society for Music Information Retrieval
- Vol. 6 (1) , 186-202
Open AccessA Dataset of Norwegian Hardanger Fiddle Recordings with Precise Annotation of Note and Beat Onsets
The Hardanger fiddle is a variety of the violin used in the folk music of the western and central part of southern Norway. This paper presents a dataset of several hours of recordings of Hardanger fiddle music, with note annotations of onsets, offsets and pitches, provided by the performers themselves. A subset has also been annotated with beat onset positions by the performer as well as three expert musicians. The complexity of the music genre—polyphonic, highly ornamented and with a very irregular pulsation, among other aspects—motivated the design of a new annotation software adapted to these particular needs. Beat annotation in MIR is typically recorded as positions in seconds, without explicit connection with actual musical events. In the context of music where the rhythm is carried by the melodic instrument alone, a more reliable definition of beat onsets consists in associating them with the onsets of the notes that represent the start of each beat. This latter definition of beat onsets reflects that beats are generated from within the flow of played melodic-rhythmic events, which implies that the spacing of beats may be shifting and irregular. This motivated the design of a new method for beat annotation in Hardanger fiddle music based on a selection of notes in the note annotation. Comparisons between annotators through alignment—integrated in the interface—enable them to eventually correct their annotations or observe alternative valid interpretations of any given excerpt. After dedicating a part of the note annotation dataset to the training of a machine learning model, for the task of assessing both note pitch and onset time, an F1 score of 87% can be reached. The beat annotation dataset demonstrates the necessity of developing new beat trackers adapted to Hardanger fiddle music. The dataset as well as the annotation software is made publicly available.
- Open Access
- 1 January 2023
- Published by SAGE Publications in Music & Science
- Vol. 6
Mental Effort and Expressive Interaction in Expert and Student String Quartet Performance
Resilience to changes in performance environments is a hallmark of expertise in music performance. Research has shown that skilled non-expert ensemble musicians maintain synchronization when visual interaction between them is disrupted, but that the quality of their expressive body motion changes. Our study extended these findings by testing how an expert string quartet responds to playing conditions that disrupt visual contact. We tested for potential effects on the musicians’ expressive head motion, sound quality, and mental effort. The Danish String Quartet (DSQ), a world-class classical ensemble, performed an excerpt from a Haydn piece five times without an audience, then once for an audience of about 20 people. During the performances without audience, their seating configuration was manipulated to disrupt their audiovisual interaction. Audio, head motion, eye-tracking, and pupillometry data were collected. The DSQ's data were compared to data from a student quartet who completed the same experiment. Our results show that the DSQ maintained the quality of their musical sound and interactive body motion across disruptive and non-disruptive conditions, but mental effort (indexed by pupil size) was greater in non-disruptive conditions. In contrast, the student quartet moved less overall than the DSQ, moved less when they could not see each other, and did not show differences in pupil size across conditions. The quartets spent a similar percentage of performance time watching their co-performers. These findings suggest that the quality of audio and visual components of the DSQ's performance do not require visual interaction to maintain; however, these musicians do interact visually when given the opportunity. This visual interaction stimulates greater mental effort, perhaps reflecting increased social engagement.
- 1 January 2023
- Published by SAGE Publications in Music & Science
- Vol. 6
Open AccessMental Effort and Expressive Interaction in Expert and Student String Quartet Performance
Resilience to changes in performance environments is a hallmark of expertise in music performance. Research has shown that skilled non-expert ensemble musicians maintain synchronization when visual interaction between them is disrupted, but that the quality of their expressive body motion changes. Our study extended these findings by testing how an expert string quartet responds to playing conditions that disrupt visual contact. We tested for potential effects on the musicians’ expressive head motion, sound quality, and mental effort. The Danish String Quartet (DSQ), a world-class classical ensemble, performed an excerpt from a Haydn piece five times without an audience, then once for an audience of about 20 people. During the performances without audience, their seating configuration was manipulated to disrupt their audiovisual interaction. Audio, head motion, eye-tracking, and pupillometry data were collected. The DSQ's data were compared to data from a student quartet who completed the same experiment. Our results show that the DSQ maintained the quality of their musical sound and interactive body motion across disruptive and non-disruptive conditions, but mental effort (indexed by pupil size) was greater in non-disruptive conditions. In contrast, the student quartet moved less overall than the DSQ, moved less when they could not see each other, and did not show differences in pupil size across conditions. The quartets spent a similar percentage of performance time watching their co-performers. These findings suggest that the quality of audio and visual components of the DSQ's performance do not require visual interaction to maintain; however, these musicians do interact visually when given the opportunity. This visual interaction stimulates greater mental effort, perhaps reflecting increased social engagement.
- 1 January 2021
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 149 (1) , 723-736
Computational localization of attack regions through a direct observation of the audio waveform
This article addresses the computational estimation of attack regions in audio recordings. Previous attempts to do so were based on the reduction of the audio waveform into an envelope curve, which decreases its temporal resolution. The proposed approach detects the attack region directly from the audio waveform. The attack region is modeled as a line starting from a low-amplitude point and intersecting one of the local maxima according to two principles: (1) maximizing the slope, while favoring, at the same time, a higher peak if the slope remains only slightly lower and (2) dismissing initial attack regions of relatively low amplitude. The attack start position is fine-tuned by intersecting the attack slope with the audio waveform. The proposed method precisely pinpoints the attack region in cases where it is unambiguously observable from the waveform itself. In such cases, previous methods selected a broader attack region due to the loss of temporal resolution. When attack regions are less evident, the proposed method's estimation remains within the range of results provided by other methods. Applied to the prediction of judgments of P-center localization [Danielsen, Nymoen, Anderson, Câmara, Langerød, Thompson, and London, J. Exp. Psychol. Hum. Percept. Perform. 45, 402–418 (2019)], the proposed method shows a significant increase in precision, at the expense of recall.
- 1 January 2021
- Published by Acoustical Society of America (ASA) in The Journal of the Acoustical Society of America
- Vol. 149 (1) , 723-736
Computational localization of attack regions through a direct observation of the audio waveform
This article addresses the computational estimation of attack regions in audio recordings. Previous attempts to do so were based on the reduction of the audio waveform into an envelope curve, which decreases its temporal resolution. The proposed approach detects the attack region directly from the audio waveform. The attack region is modeled as a line starting from a low-amplitude point and intersecting one of the local maxima according to two principles: (1) maximizing the slope, while favoring, at the same time, a higher peak if the slope remains only slightly lower and (2) dismissing initial attack regions of relatively low amplitude. The attack start position is fine-tuned by intersecting the attack slope with the audio waveform. The proposed method precisely pinpoints the attack region in cases where it is unambiguously observable from the waveform itself. In such cases, previous methods selected a broader attack region due to the loss of temporal resolution. When attack regions are less evident, the proposed method's estimation remains within the range of results provided by other methods. Applied to the prediction of judgments of P-center localization [Danielsen, Nymoen, Anderson, Câmara, Langerød, Thompson, and London, J. Exp. Psychol. Hum. Percept. Perform. 45, 402–418 (2019)], the proposed method shows a significant increase in precision, at the expense of recall.
- Open Access
- 26 November 2019
- Published by MDPI AG in Applied Sciences
- Vol. 9 (23) , 5121
Tempo and Metrical Analysis by Tracking Multiple Metrical Levels Using Autocorrelation
We present a method for tempo estimation from audio recordings based on signal processing and peak tracking, and not depending on training on ground-truth data. First, an accentuation curve, emphasizing the temporal location and accentuation of notes, is based on a detection of bursts of energy localized in time and frequency. This enables the detection of notes in dense polyphonic texture, while ignoring spectral fluctuation produced by vibrato and tremolo. Periodicities in the accentuation curve are detected using an improved version of autocorrelation function. Hierarchical metrical structures, composed of a large set of periodicities in pairwise harmonic relationships, are tracked over time. In this way, the metrical structure can be tracked even if the rhythmical emphasis switches from one metrical level to another. This approach, compared to all the other participants to the Music Information Retrieval Evaluation eXchange (MIREX) Audio Tempo Extraction competition from 2006 to 2018, is the third best one among those that can track tempo variations. While the two best methods are based on machine learning, our method suggests a way to track tempo founded on signal processing and heuristics-based peak tracking. Moreover, the approach offers for the first time a detailed representation of the dynamic evolution of the metrical structure. The method is integrated into MIRtoolbox, a Matlab toolbox freely available.
- 26 November 2019
- Published by MDPI AG in Applied Sciences
- Vol. 9 (23) , 5121
Open AccessTempo and Metrical Analysis by Tracking Multiple Metrical Levels Using Autocorrelation
We present a method for tempo estimation from audio recordings based on signal processing and peak tracking, and not depending on training on ground-truth data. First, an accentuation curve, emphasizing the temporal location and accentuation of notes, is based on a detection of bursts of energy localized in time and frequency. This enables the detection of notes in dense polyphonic texture, while ignoring spectral fluctuation produced by vibrato and tremolo. Periodicities in the accentuation curve are detected using an improved version of autocorrelation function. Hierarchical metrical structures, composed of a large set of periodicities in pairwise harmonic relationships, are tracked over time. In this way, the metrical structure can be tracked even if the rhythmical emphasis switches from one metrical level to another. This approach, compared to all the other participants to the Music Information Retrieval Evaluation eXchange (MIREX) Audio Tempo Extraction competition from 2006 to 2018, is the third best one among those that can track tempo variations. While the two best methods are based on machine learning, our method suggests a way to track tempo founded on signal processing and heuristics-based peak tracking. Moreover, the approach offers for the first time a detailed representation of the dynamic evolution of the metrical structure. The method is integrated into MIRtoolbox, a Matlab toolbox freely available.