Chris Kiefer

Opera has historically been a site for developing new and emerging technologies. More recently, opera has proven a rich environment for investigating future scenarios involving human-robot interaction and exploring the potential of robot performers. The explosion of enthusiasm for AI and the ubiquity of machine learning tools have created numerous possibilities for generating creative content in music, poetry, images, and choreography. This article outlines efforts to explore the creative possibilities of robots and AI through a dedicated workshop with researchers from diverse fields. Our goal is to begin thinking beyond the human to investigate what robot performers might look and behave like when designed from a relational rather than a representational approach. We provide a brief history of robots on stage, including the authors’ prior work developing original performances featuring robots and human performers. We describe our iterative design approach and low- fi prototypes, summarizing key insights from the workshop. Finally, we analyze the workshop outcomes and consider the trade-offs of working with existing robots and the possibilities for working with open, configurable systems.

Feedback is a technique that has been used in musical performance since the advent of electricity. From the early cybernetic explorations of Bebe and Louis Barron, through the screaming sound of Hendrix’s guitar, to the systems design of David Tudor or Nic Collins, we find the origins of feedback in music being technologically and aesthetically diverse. Through interviews with participants in a recent Feedback Musicianship Network symposium, this paper seeks to investigate the contemporary use of this technique and explore how key protagonists discuss the nature of their practice. We see common concepts emerging in these conversations: agency, complexity, coupling, play, design and post-humanism. The paper presents a terminological and ideological framework as manifested at this point in time, and makes a theoretical contribution to the understanding of the rationale and potential of this technological and compositional approach.

This paper reports on early advances in the design of a browser-based ecosystem for creating new live coding languages, optimal for audio synthesis, machine learning, and machine listening. We present the rationale and challenges when applying the Web Audio API to the design of a high-performance signal synthesis engine, using an AudioWorklet-based solution and refactoring our digital signal processing library Maximilian.js. Furthermore, we contribute with the latest advances in Sema, a new user-friendly playground that integrates the signal engine to empower the live coding community to design their own idiosyncratic languages and interfaces. The evaluation shows that the system runs with high reliability and efficiency and low latency.

The benefits of ensemble performance are well recognized; notation supports group performance, but cuts in music education and changing musical cultures mean that notation is increasingly perceived as a barrier to entry. In an extended participatory design (PD) project, we co-designed and developed a software system for networked notation called Syncphonia with the aim of enhancing access to and experience of notation-based ensemble performance. In previous work, our formal evaluation and informal observations and feedback revealed a wide range of benefits. In this article, we are concerned with articulating the knowledge generated and insights gained through this extended PD process. To do so, we employ a framework for systematic reflection that has been designed to support investigation into the tacit knowledge generated in participatory design. Through this method, we focus inward and share three insights into the value of networked notation in contemporary musical cultures; we also look outward and articulate five approaches to PD with musical ensembles that might benefit others adopting this rich research method. A pluralistic and inclusive vision of notation is espoused and speculation is submitted that a dynamic, networked notation might ameliorate the boundaries between composing, improvising and performing to the benefit of all three.

As augmented reality (AR) quickly evolves with new technological practice, there is a growing need to question and reevaluate its potential as a medium for creative expression. The authors discuss AR within computational art, framed within AR as a medium, AR aesthetics and applications. The Forum for Augmented Reality Immersive Instruments (ARImI), a two-day event on AR, highlights both possibilities and fundamental concerns for continuing artworks in this field, including visual bias, sensory modalities, interactivity and performativity. The authors offer a new AR definition as real-time computationally mediated perception.

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with the generation of multi-timbre audio patterns, and little testing has been done on scalability to longer patterns required for audio. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined.

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music and procedural audio but has yet to be explored. Two novel methods of sound synthesis based on conceptors are introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. Conceptillators are trainable, pitch-controlled oscillators for harmonically rich waveforms, commonly used in a variety of sound synthesis applications. Both systems can exploit conceptor pattern morphing, boolean logic and manipulation of RNN dynamics, enabling new creative sonic possibilities. Experiments reveal how RNN runtime parameters can be used for pitch-independent timestretching and for precise frequency control of cyclic waveforms. They show how these techniques can create highly malleable sound synthesis models, trainable using short sound samples. Limitations are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where exponential rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors; future possibilities and research questions are outlined, including possibilities in generative sound.