Multi‐Modal Instrument Performances (MMIP): A Musical Database
Open Access
- 10 April 2025
- journal article
- Published by Wiley in Computer Graphics Forum
- Vol. 44 (2)
Abstract
Musical instrument performances are multimodal creative art forms that integrate audiovisual elements, resulting from musicians' interactions with instruments through body movements, finger actions, and facial expressions. Digitizing such performances for archiving, streaming, analysis, or synthesis requires capturing every element that shapes the overall experience, which is crucial for preserving the performance's essence. In this work, following current trends in large‐scale dataset development for deep learning analysis and generative models, we introduce the Multi‐Modal Instrument Performances (MMIP) database (https://mmip.cs.ucy.ac.cy). This is the first dataset to incorporate synchronized high‐quality 3D motion capture data for the body, fingers, facial expressions, and instruments, along with audio, multi‐angle videos, and MIDI data. The database currently includes 3.5 hours of performances featuring three instruments: guitar, piano, and drums. Additionally, we discuss the challenges of acquiring these multi‐modal data, detailing our approach to data collection, signal synchronization, annotation, and metadata management. Our data formats align with industry standards for ease of use, and we have developed an open‐access online repository that offers a user‐friendly environment for data exploration, supporting data organization, search capabilities, and custom visualization tools. Notable features include a MIDI‐to‐instrument animation project for visualizing the instruments and a script for playing back FBX files with synchronized audio in a web environment.Keywords
Funding Information
- Horizon 2020 Framework Programme (739578)
- Deputy Ministry of Research, Innovation and Digital Policy
- Horizon 2020 Framework Programme (101178362, 101061303)
This publication has 36 references indexed in Scilit:
- Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild2024
- InterGen: Diffusion-Based Multi-human Motion Generation Under Complex InteractionsInternational Journal of Computer Vision, 2024
- One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer2023
- FLAG3D: A 3D Fitness Activity Dataset with Language Instruction2023
- ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation2023
- Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicineEBioMedicine, 2023
- Three dimensional motion capture applied to violin playing: A study on feasibility and characterization of the motor strategyComputer Methods and Programs in Biomedicine, 2017
- The KIT Motion-Language DatasetBig Data, 2016
- Estimation of Guitar Fingering and Plucking Controls Based on Multimodal Analysis of Motion, Audio and Musical Score2016
- The Sense of Ensemble: a Machine Learning Approach to Expressive Performance Modelling in String QuartetsJournal of New Music Research, 2014