Atau Tanaka

With the emergence of technologies for collecting data for the body, such as motion tracking and biosignal sensing, there has been an interest in using these technologies in performing arts. In such performance settings, this data has been mapped to different types of output, such as audio-visuals. In this chapter, we discuss the intersection of embodied interaction, performing arts and audio-visual practice. In particular, we will present two lines of work, one exploring audio-visuals in contemporary dance (the Moving Digits project) and another combining gesture, electronic music, and visuals (works by Atau Tanaka). From the analysis of these two lines of work and related projects, we map out the area of embodied interaction in performative audio-visual practice and identify relevant categories.

We present a participatory co-design cycle that studied synaesthesia, a cross-sensory phenomenon where one stimulus automatically triggers another perception. We worked with a community of synaesthetes to develop and evaluate an interactive, immersive technology to depict and c dommunicate syaesthetic experience. By working with qualitative methods to understand experience, we accomodate the highly individual nature and support otherwise limited sensory vocabulary and lack of shared understanding of the condition. The first study involved a series of participatory workshops that explored text, creative tools, interactive technologies, and sensory materials in conveying the synaesthetic experience. Insights from these workshops informed the development of the Somatic Drawing Tool, a VR drawing system that allows users to depict their experiences. Within the tool, we deployed a the 3D Body Sheet, an immersive extension of techniques drawn from soma design. We engaged users in an initial evaluation of the system and took on board their feedback in a subsequent cycle of iterative development. This resulted in a bespoke system that could be personalised to diverse types of individual synaesthetic experience. This enables nuanced articulation, providing a supportive environment for externalising and reflecting sensations. These approaches for capturing the experiential nature of embodied sensory perception has potential to be applied in broader research in cross-modal and multisensory interaction.

This paper investigates sound and music interactions arising from the use of electromyography (EMG) to instrumentalise signals from muscle exertion of the human body. We situate EMG within a family of embodied interaction modalities, where it occupies a middle ground, considered as a “signal from the inside” compared with external observations of the body (e.g., motion capture), but also seen as more volitional than neurological states recorded by brain electroencephalogram (EEG). To understand the messiness of gestural interaction afforded by EMG, we revisit the phenomenological turn in HCI, reading Paul Dourish’s work on the transparency of “ready-to-hand” technologies against the grain of recent posthumanist theories, which offer a performative interpretation of musical entanglements between bodies, signals, and representations. We take music performance as a use case, reporting on the opportunities and constraints posed by EMG in workshop-based studies of vocal, instrumental, and electronic practices. We observe that across our diverse range of musical subjects, they consistently challenged notions of EMG as a transparent tool that directly registered the state of the body, reporting instead that it took on “present-at-hand” qualities, defamiliarising the performer’s own sense of themselves and reconfiguring their embodied practice.

We present an interactive modular system built in Cycling ‘74 Max and interfaced with Grame’s FAUST for the purpose of analyzing, processing and mapping electrophysiological signals to sound. The system architecture combines an understanding of domain-specific (biophysiological) signal processing techniques with a flexible, modular and user-friendly interface. We explain our design process and decisions towards artistic usability, while maintaining a clear electrophysiological data flow. The system allows users to customize and experiment with different configurations of sensors, signal processing and sound synthesis algorithms, and has been tested in a range of different musical settings from user studies to concerts with a diverse range of musicians.

This article presents a custom system combining hardware and software that senses physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.