Towards Learning to Play Piano with Dexterous Hands and Touch
Open Access
- 23 October 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10410-10416
Abstract
As Liszt once said “(a virtuoso) must call up scent and blossom, and breathe the breath of life”, a virtuoso plays the piano with passion, poetry, and extraordinary technical ability. Hence, piano playing, being a task that is quintessentially human, becomes a hallmark for roboticians and artificial intelligence researchers to pursue. In this paper, we advocate an end-to-end reinforcement learning (RL) paradigm to demonstrate how an agent can learn directly from machine-readable music score to play the piano with touch-augmented dexterous hands on a simulated piano. To achieve the desired tasks, we design useful touch- and audio-based reward functions and a series of tasks. Empirical results show that the RL agent can not only find the correct key position but also deal with the various rhythmic, volume, and fingering requirements. As a result, the agent demonstrates its effectiveness in playing simple pieces that have different musical requirements which show the potential of leveraging reinforcement learning approach for the piano playing tasks.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and ForceSensors, 2017
- Finger tracking for piano playing through contactless sensor system: Signal processing and data training using artificial neural network2017
- Piano playing robot2016
- Development on an intelligent music score input system — Applied for the piano robot2016
- Force control for the fingers of the piano playing robot — A gain switched approach2015
- Intelligent algorithm for music playing robot — Applied to the anthropomorphic piano robot control2014
- Controller design for music playing robot — Applied to the anthropomorphic piano robot2013
- Electronic piano playing robot2010
- Calculation of the Wasserstein Distance Between Probability Distributions on the LineTheory of Probability & Its Applications, 1974
- A Markovian Decision ProcessIndiana University Mathematics Journal, 1957