Eduardo A. L. Meneses

Singing voice synthesis involves the artificial generation of a song given its lyrics and musical notes through computational methods. This paper describes the development and evaluation process of PATRICIA, a proof-of-concept prototype of a system that performs real-time singing voice synthesis (SVS) for the Brazilian Portuguese language. It is the first singing synthesizer specifically designed for this idiom. A literature review was conducted to study the latest developments in real-time singing synthesis and to give directions for PATRICIA design and implementation. The implemented architecture is described, and, for the validation process, a series of video recordings of the system working was made as a musical demonstration. This demonstration was subjected to an evaluation by musical educators. A performance analysis with two CPU usage indicators in two different devices was also conducted. The results of the validation are discussed, and future enhancements are pointed out in order to overcome the system’s current limitations.

Based on established artistic practices focusing on spatial audio in the context of immersive sound experiences, we bring forth and discuss the concept of scenophony – following Gilles Malatray’s description of the word ‘scénophonie’. Derived from the term scenography, scenophony refers to the practice of the (scenic) space, primarily considering its acoustic situation (phonic) as perceived by the audience. Scenophony is the design and realization of the sound space as a whole: a perception-led creative practice at the intersection of sound design, audio art, musical composition, acoustics, psychoacoustics, audio engineering, and user experience design. The main object of this article is to define the role of scenophony and its involvement in the creation and overall reception of a sonic experience within or as a scenic space, as well as its potential applications and benefits in the context of immersive storytelling. Moreover, we propose a paradigm shift from the common representations used in most current sound spatialization tools, generally assuming a single, ideal, egocentric, sweet-spot-oriented approach – mostly aimed at VR/AR/MR applications – as evidenced by both the graphical (UI) representation of Euclidean space as well as the preferred, often single, audio spatialization technique proposed. The hypothesis is that such egocentric graphical representations of some sound spatialization tools may limit how sound and music are diffused, as well as affect how sonic scenes are designed and listened to. In contrast to such representations, especially in the context of collective and heterogeneous sound experiences, we propose that an allocentric approach to spatial sound design, grounded in the concept of scenophony, offers new ways to create stories through immersive sound.

This hands-on class will allow artists to use open-source tools to create interactive and immersive experiences. These tools have been created and incubated at the Society for Arts and Technology (SAT), a unique non-profit organization in Canada whose mission is to democratize technologies to enable people to experience and author multisensory immersions. During the class we invite participants to use their favorite software on platforms they are already familiar with, to interface with our tools. The toolset will include transmission protocols, video mapping tools, sound spatialization software, and gestural control using pose detection. The class will be organized in two parts: a presentation of the tools and context involving the development and applications, and a hands-on session with an ephemeral immersive space. This event is designed for art researchers, artists, designers, content creators, and other creatives interested in creating immersive spaces using research-developed tools. Participants will learn how to employ open-source tools for different artistic tasks so that they will be able to deploy their own immersive spaces after the class.

Computing technologies have opened up a myriad of possibilities for expanding the sonic capabilities of acoustic musical instruments. Musicians nowadays employ a variety of rather inexpensive, wireless sensor-based systems to obtain refined control of interactive musical performances in actual musical situations like live music concerts. It is essential though to clearly understand the capabilities and limitations of such acquisition systems and their potential influence on high-level control of musical processes. In this study, we evaluate one such system composed of an inertial sensor (MetaMotionR) and a hexaphonic nylon guitar for capturing strumming gestures. To characterize this system, we compared it with a high-end commercial motion capture system (Qualisys) typically used in the controlled environments of research laboratories, in two complementary tasks: comparisons of rotational and translational data. For the rotations, we were able to compare our results with those that are found in the literature, obtaining RMSE below 10° for 88% of the curves. The translations were compared in two ways: by double derivation of positional data from the mocap and by double integration of IMU acceleration data. For the task of estimating displacements from acceleration data, we developed a compensative-integration method to deal with the oscillatory character of the strumming, whose approximative results are very dependent on the type of gestures and segmentation; a value of 0.77 was obtained for the average of the normalized covariance coefficients of the displacement magnitudes. Although not in the ideal range, these results point to a clearly acceptable trade-off between the flexibility, portability and low cost of the proposed system when compared to the limited use and cost of the high-end motion capture standard in interactive music setups.