Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks
Preprint
Abstract
Generating high-quality piano audio from video requires precise synchronization between visual cues and musical output, ensuring accurate semantic and temporal alignment.However, existing evaluation datasets do not fully capture the intricate synchronization required for piano music generation. A comprehensive benchmark is essential for two primary reasons: (1) existing metrics fail to reflect the complexity of video-to-piano music interactions, and (2) a dedicated benchmark dataset can provide valuable insights to accelerate progress in high-quality piano music generation. To address these challenges, we introduce the CoP Benchmark Dataset-a fully open-sourced, multimodal benchmark designed specifically for video-guided piano music generation. The proposed Chain-of-Perform (CoP) benchmark offers several compelling features: (1) detailed multimodal annotations, enabling precise semantic and temporal alignment between video content and piano audio via step-by-step Chain-of-Perform guidance; (2) a versatile evaluation framework for rigorous assessment of both general-purpose and specialized video-to-piano generation tasks; and (3) full open-sourcing of the dataset, annotations, and evaluation protocols. The dataset is publicly available at https://github.com/acappemin/Video-to-Audio-and-Piano, with a continuously updated leaderboard to promote ongoing research in this domain.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized SoundsInternational Journal of Computer Vision, 2026
- Video-Guided Foley Sound Generation with Multimodal Controls2025
- Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS ControlProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Temporally Aligned Audio for Video with Autoregression2025
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models2023
- Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models2022