Bartłomiej Mróz

This paper describes an architecture design process for Networked Music Performance (NMP) platform for medium-sized conducted music ensembles, based on remote rehearsals of Academic Choir of Gdansk University of Technology. The issues of real-time remote communication, in-person music performance, and NMP are described. Three iterative steps defining and extending the architecture of the NMP platform with additional features to enhance its utility in remote rehearsals are presented. The first iteration uses a regular video conferencing platform, the second iteration uses dedicated NMP devices and tools, and the third iteration adds video transmission and utilizes professional low-latency audio and video workstations. For each iteration, the platform architecture is defined and deployed with simultaneous usability tests. Its strengths and weaknesses are identified through qualitative and quantitative measurements - statistical analysis shows a significant improvement in rehearsal quality after each iteration. The final optimal architecture is described and concluded with guidelines for creating NMP systems for said music ensembles.

The aim of this paper is three-fold. First, the basics of binaural and ambisonic techniques are briefly presented. Then, details related to audio-visual recordings of a remote performance of the Academic Choir of the Gdansk University of Technology are shown. Due to the COVID-19 pandemic, artists had a choice, namely, to stay at home and not perform or stay at home and perform. In fact, staying at home brought in the possibility of creating and developing art at home while working online. During the first months of lock-down, the audience was satisfied with music performances that were fairly far from the typical experience of a real concert hall. Then, more advanced technology was brought to facilitate joint rehearsal and performance of better quality, including multichannel sound and spatialization. At the same time, spatial music productions benefited from the disadvantage of remote rehearsal by creating immersive experiences for the audience based on ambisonic and binaural techniques. Finally, subjective tests were prepared and performed to observe performers’ attention behavior divided between the conductor and music notation in the network-like environment. To this end, eye-tracking technology was employed. This aspect is related to the quality of experience (QoE), which in the performance area–and especially in remote mode–is essential.

This paper describes a method for planning, recording, and post-production of six-degrees-of-freedom audio recorded with multiple 3$^{\mathrm{rd}}$ order Ambisonic microphone arrays. The description is based on the example of recordings conducted in August 2020 with the Poznan Philharmonic Orchestra using 30 units of Zylia ZM-1S. A convenient way to prepare and organize such a big project is proposed – this involves details of stage planning, collecting needed equipment, management, and running the recording. Additionally, the placement of 2D and 360 cameras as well as the design of moving camera paths are described. Two use cases of the collected Ambisonic audio are proposed. The first one is a production of a virtual reality application using Unreal Engine 4 and Wwise. The second one uses a combination of the virtual scene from the previously mentioned application and a footage from a mobile, moving camera to create a so-called “walkthrough movie”. The proposed approach for six-degrees-of-freedom Ambisonic interpolation was evaluated on listening tests. Those tests demonstrate that it is possible to recreate a navigable 3D audio taking into account Ambisonic only input and given assumptions for microphone placement.