HGHannes Gamper

Hannes Gamper

*This is an algorithmically generated author record based on publicly available information.

Hannes Gamper has a total of 17 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Make Some Noise: Towards LLM audio reasoning and generation using sound tokens

    by Shivam Mehta, Nebojsa Jojic, Hannes Gamper

    Integrating audio comprehension and generation into large language models (LLMs) remains challenging due to the continuous nature of audio and the resulting high sampling rates. Here, we introduce a novel approach that combines Variational Quantization with Conditional Flow Matching to convert audio into ultra-low bitrate discrete tokens of 0.23kpbs, allowing for seamless integration with text tokens in LLMs. We fine-tuned a pretrained text-based LLM using Low-Rank Adaptation (LoRA) to assess its effectiveness in achieving true multimodal capabilities, i.e., audio comprehension and generation. Our tokenizer outperforms a traditional VQ-VAE across various datasets with diverse acoustic events. Despite the substantial loss of fine-grained details through audio tokenization, our multimodal LLM trained with discrete tokens achieves competitive results in audio comprehension with state-of-the-art methods, though audio generation is poor. Our results highlight the need for larger, more diverse datasets and improved evaluation metrics to advance multimodal LLM performance.

  • Open Access

    Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment

    by Benjamin Stahl, Hannes Gamper

    In this paper, we investigate distillation and pruning methods to reduce model size for non-intrusive speech quality assessment based on self-supervised representations. Our experiments build on XLS-R-SQA, a speech quality assessment model using wav2vec 2.0 XLS-R embeddings. We retrain this model on a large compilation of mean opinion score datasets, encompassing over 100,000 labeled clips. For distillation, using this model as a teacher, we generate pseudo-labels on unlabeled degraded speech signals and train student models of varying sizes. For pruning, we use a data-driven strategy. While data-driven pruning performs better at larger model sizes, distillation on unlabeled data is more effective for smaller model sizes. Distillation can halve the gap between the baseline's correlation with ground-truth MOS labels and that of the XLS-R-based teacher model, while reducing model size by two orders of magnitude compared to the teacher model.

  • Open Access

    Multi-label audio classification with a noisy zero-shot teacher

    by Sebastian Braun, Hannes Gamper

    We propose a novel training scheme using self-label correction and data augmentation methods designed to deal with noisy labels and improve real-world accuracy on a polyphonic audio content detection task. The augmentation method reduces label noise by mixing multiple audio clips and joining their labels, while being compatible with multiple active labels. We additionally show that performance can be improved by a self-label correction method using the same pretrained model. Finally, we show that it is feasible to use a strong zero-shot model such as CLAP to generate labels for unlabeled data and improve the results using the proposed training and label enhancement methods. The resulting model performs similar to CLAP while being an efficient mobile device friendly architecture and can be quickly adapted to unlabeled sound classes.

  • Open Access

    Spatio-Temporal Windowing for Encoding Perceptually Salient Early Reflections in Parametric Spatial Audio Rendering

    by Tobias Jú Terbock, Fabian Brinkmann, Hannes Gamper, Nikunj Raghuvanshi, Stefan Weinzierl

    Parametric spatial audio rendering aims to provide perceptually convincing audio cues that are agnostic to the playback system to enable the acoustic design of games and virtual reality. The authors propose an algorithm for detecting perceptually important reflections from spatial room impulse responses. First, a parametric representation of the sound field is derived based on perceptually motivated spatio-temporal windowing, followed by a second step that estimates the perceptual salience of...


  • Open Access

    Interactive and Immersive Auralization

    by Nikunj Raghuvanshi, Hannes Gamper

    Real-time auralization is essential in virtual reality (VR), gaming, and architecture to enable an immersive audio-visual experience. The audio rendering must be congruent with visual feedback and respond with minimal delay to interactive events and user motion. The wave nature of sound poses critical challenges for plausible and immersive rendering and leads to enormous computational costs. These costs have only increased as virtual scenes have progressed away from enclosures toward complex, city-scale scenes that mix indoor and outdoor areas. However, hard real-time constraints must be obeyed while supporting numerous dynamic sound sources, frequently within a tightly limited computational budget. In this chapter, we provide a general overview of VR auralization systems and approaches that allow them to meet such stringent requirements. We focus on the mathematical foundation, perceptual considerations, and application-specific design requirements of practical systems today, and the future challenges that remain.


  • Open Access

    Large-scale simulation of bird localization systems in forests with distributed microphone arrays

    by Shoken Kaneko, Hannes Gamper

    Acoustic wildlife monitoring systems are important tools for capturing information about animal habitation in ecosystems. Previous work has demonstrated the effectiveness of audio-based bird localization techniques. However, few studies have investigated the performance and robustness of distributed systems in large forests. Here, the performance of distributed microphone arrays for localizing birds is examined by simulating forest scenes with added reverberation, ambient noise, and measurement errors. The simulation revealed the importance of the signal-to-noise ratio and the spectral weighting in the localization algorithm. These results may guide the design of large-scale wildlife monitoring systems and suggest promising directions for further improvements.


  • Open Access

    Predicting word error rate for reverberant speech

    by Hannes Gamper, Dimitra Emmanouilidou, Sebastian Braun, Ivan J. Tashev

    Reverberation negatively impacts the performance of automatic speech recognition (ASR). Prior work on quantifying the effect of reverberation has shown that clarity (C50), a parameter that can be estimated from the acoustic impulse response, is correlated with ASR performance. In this paper we propose predicting ASR performance in terms of the word error rate (WER) directly from acoustic parameters via a polynomial, sigmoidal, or neural network fit, as well as blindly from reverberant speech samples using a convolutional neural network (CNN). We carry out experiments on two state-of-the-art ASR models and a large set of acoustic impulse responses (AIRs). The results confirm C50 and C80 to be highly correlated with WER, allowing WER to be predicted with the proposed fitting approaches. The proposed non-intrusive CNN model outperforms C50-based WER prediction, indicating that WER can be estimated blindly, i.e., directly from the reverberant speech samples without knowledge of the acoustic parameters.

  • Characterizing acoustic environments using spherical loudspeaker and microphone arrays

    by Hannes Gamper, Keith Godin, Nikunj Raghuvanshi, Ivan J. Tashev

    Room acoustic parameters, including the reverberation time (T60) and the direct-to-reverberant ratio (DRR), quantitatively describe the behaviour of an acoustic environment. These parameters are typically derived from an acoustic impulse response (AIR) obtained using an omnidirectional sound source and an omnidirectional receiver. However, the acoustic interaction between a source with a complex radiation pattern, for example, a human talker, and a receiver with a complex directivity pattern, for example, a human listener, cannot be accurately described by an omnidirectional AIR or the acoustic parameters derived from it. Here, we propose characterizing acoustic environments while taking the source and receiver directionality into account. A spherical loudspeaker array allows modelling a source with an arbitrary radiation pattern. Analogously, a spherical microphone can be used to describe a receiver with an arbitrary directivity pattern. By combining spherical loudspeaker and microphone arrays, a multiple-input multiple-output (MIMO) AIR of an acoustic environment can be measured to simulate sources and receivers with arbitrary directivities and estimate acoustic parameters accordingly.

  • Improving the perception of a sound source's polar angle in mixed reality

    by Hannes Gamper, Ivan J. Tashev

    Mixed reality applications blend real and virtual scenes. To render virtual objects in a scene, the rendering system needs to accurately control their perceived location. In the acoustic domain, the location of a sound source is often given in interaural coordinates, i.e., as the lateral and polar angle and distance relative to the midpoint of the listener's interaural axis. This description is useful as it allows separating the effect of various perceptual cues on each interaural spatial dimension. Prior research has shown that the human perception of a sound source's polar angle, i.e., the angle of rotation about the interaural axis, tends to be less accurate than the perception of its lateral angle, i.e., the angle off the median plane. When rendering virtual sound sources, listeners often confuse locations above and below the horizontal plane or in the front and in the back. Here, we review cues that affect the perception of polar angle as well as approaches to improve the accuracy of rendering the polar angle of a virtual sound source in mixed reality applications.