Gou Koutaki

RoboSax Melody Slot Machine is a mobile-to-acoustic performance system that connects an interactive tablet interface to live saxophone performance. Participants select melodic fragments on an iPad-based Melody Slot Machine, where related musical phrases are generated through GTTM-based melody morphing rather than arbitrary recombination. The selected melody is transmitted as MIDI to RoboSax, a robotic saxophone mechanism that actuates the instrument’s fingerings, while a human performer provides breath, tonguing, phrasing, dynamics, and tone color. This division of roles allows participants without instrumental training to influence the musical structure of a live performance while preserving the embodied expressiveness of acoustic wind performance. For SIGGRAPH Appy Hour, the work presents a participatory experience in which mobile interaction becomes immediately audible as live acoustic sound. By combining structurally coherent melody variation, audience-controlled interaction, and human–robot performance, RoboSax Melody Slot Machine explores how mobile music apps can extend beyond the screen and become part of a shared performative space.

This paper proposes an acoustic analysis system to help improve saxophone performance skills. The system combines direct support for performance movements by a robot with indirect support by presenting performance information. By sensing the performance audio and performing real-time acoustic analysis, the system presents the learner with information about their performance and their playing habits. The performance information presented to the learner includes pitch, volume, and playing timing. For performance habit analysis, a Markov model with pitch as the state and an internal probability parameter that indicates the quality of the performance evaluation as the pitch transitions are defined. In the experiment, we conducted a pilot study targeting experienced saxophone players and a beginner saxophone player to verify the effectiveness of the proposed system. The experiment showed that the MAE of the played pitch was significantly reduced by using the proposed system.

We propose a semi-automatic artificial lips control device that allows a human performer to produce sound on a brass instrument without the need to vibrate their own lips. The device integrates position control that presses artificial lips toward the mouthpiece and aperture control via wire traction, together with a pre-calibrated motor table and acoustic feedback for pitch stabilization. In evaluations using a euphonium, we verified timbre, pitch range, and pitch stabilization, including harmonic modes. The results showed that the harmonic structure of tones produced by a human using the device can be comparable to those produced by a human player in the conventional manner. Pitch-range and pitch-stabilization tests confirmed that the system can generate practical musical intervals and achieve reliable harmonic mode changes. Furthermore, real-time acoustic feedback improved pitch stability during performance. These findings demonstrate that, rather than fully automating human performance, the proposed system provides a compact and reproducible framework for controllable brass sound generation and pitch stabilization using only three actuators.

With the rapid advancements in autonomous driving in recent years, the importance of HD (High Definition) map referenced during autonomous driving has also increased. The mainstream method for creating HD maps involves using data collected from MMS (Mobile Mapping System) equipped with multiple sensors. Accurate fusion of data from multiple sensors is required. Typically, calibration between sensors is done by simultaneously measuring the same object. However, some sensors on MMS have non-overlapping FoV (Field of View), making it difficult to accurately calibrate and fuse their data. In this study, we focus on the fusion of data from a camera and LiDAR with non-overlapping FoVs. By estimating the relative positional information between the camera and LiDAR using two same calibration target by ’Total station’, we performed data fusion. Originally, the total station are used for civil surveying, but we show that they can be used for camera and LiDAR calibration in computer vision. Specifically, two boards of the same make are placed in the field of view of the camera and LiDAR, and the relative positions of the two boards are matched by a total station, enabling highly accurate camera and LiDAR calibration even when the fields of view do not overlap. In simulation experiments, a translational error of 15.28 mm was achieved, which is approximately 4 mm smaller than the conventional target-based method.

Grasping and manipulating transparent objects with a robot is a challenge in robot vision. To successfully perform robotic grasping, 6D object pose estimation is needed. However, transparent objects are difficult to recognize because their appearance varies depending on the background, and modern 3D sensors cannot collect reliable depth data on transparent object surfaces due to the translucent, refractive, and specular surfaces. To address these challenges, we proposed a 6D pose estimation of transparent objects for manipulation. Given a single RGB image of transparent objects, the 2D keypoints are estimated using a deep neural network. Then, the PnP algorithm takes camera intrinsics, object model size, and keypoints as inputs to estimate the 6D pose of the object. Finally, the predicted poses of the transparent object were used for grasp planning. Our experiments demonstrated that our picking system is capable of grasping transparent objects from different backgrounds. To the best of our knowledge, this is the first time a robot has grasped transparent objects from a single RGB image. Furthermore, the experiments show that our method is better than the 6D pose estimation baselines and can be generalized to real-world images.

In April 2016, a massive earthquake with a magnitude of 7.3 struck Kumamoto region, Japan, causing major devastation. One of the structures that were damaged in Kumamoto was Kumamoto Castle, a cultural asset of great significance in Japan. The stone retaining wall “ishigaki” that formed the foundation of the castle collapsed, and the superstructure was destroyed. The number of stones is estimated to be more than 70,000, and restoration work is anticipated to take more than 20 years. Since each of the stones is an important cultural asset, the broken stone structure needed to be restored to its original state in order not to lose its cultural value forever. In addition, the fallen stones need to be returned to their original positions in the ishigakis. In similar cases, non-automatic visual verification was used. However, for Kumamoto Castle, this would have been impossible because a large number of stones were displaced as a result of the collapse. The purpose of this project is to provide support for the restoration work by matching the stones that fell down after the collapse with those before the collapse using information technology, such as computer vision and optimization technologies. Specifically, we captured photographic images of the stones before and after the collapse to match them. The technical contributions of this study are as follows: (a) To estimate the scale and surface orientation of the stones, we exploit 3D model construction from the images. (b) To solve the jigsaw-puzzle-like problem of reassembling the stone fragments, we exploit the combination of a customized iterative closest point (ICP) algorithm for shape position matching and an assignment algorithm to find the best pairs of stones before and after the collapse by using the matching degree obtained from ICP. Here, only the 2D shape of the stones before the collapse can be used due to the small number of photos available. In contrast, a detailed 3D shape can be obtained from the stones after their collapse. We matched these asymmetric data in 2D and 3D to enable a comprehensive reconstruction. (c) We developed a user-friendly graphical user interface system that was used by actual masons without special knowledge. The developed system was used to match the ishigaki of a turret, Iidamaru. As a result, we succeeded in identifying 337 stones, or approximately 90% of the 370 images. These results are expected to be useful for and were used as a blueprint during actual restoration work.